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CHAPTER I 



INTRODUCTION 



Statement o f Purpos e 

It is the purpose of this paper to define the competitive computer selection 
system used by the United States Navy. Other selection systems are described, and 
the Navy's system is evaluated in relation to these other systems. 

Organization of Study 

In keeping with its expressed purpose, this thesis is divided into several 
chapters, each chapter provides the reader with a logical element from the total 
package. 

The competit ive computer selection process used by the Department of the 
Navy is highly regulated and owes its origin to the fact that the Navy is a govern- 
mental agency with a central office responsible for the selection pi*ocess. In 
Chapter II, the various controls exerted by Congress, the President, the Department 
of Defense, and the Secretary of the Navy are examined. The responsible central 
office is a very significant factor shaping the selection system. Accordingly, the 
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mission and organizational relationships of this office are defined. 

Chapter III examines the details of the selection procedures. The total cycle 
of acquiring a computer system involves two distinct and separate phases. First, 
the user conducts a system study to define the management data system and prove 
its soundness. The second part concerns the procurement actions necessary to 
bring a computer system on board. This paper will address only the latter part of 
the cycle and thus assumes that a good system study has been conducted and the 
prospective user therefore knows his true systems requirements. Chapter III is 
therefore directed toward the five major activities that involve the central selection 
office. The five activities are: Specification Review, Preparation of the Request for 
Proposal and Selection Plan, Vendor Liaison and Validation of Proposals, Evaluation 
of Proposal, and Contract Negotiation and Award. 

Chapter IV discusses the results of a survey conducted by Dr. Norman 
Scbneidewind. This survey purports to show four different catergories of users, and 
what practices are used, to select computer systems. In addition, the validation 
techniques of computer simulation and hand timing are discussed. The latter part 
of this chapter is directed toward two alternative evaluation systems- -the Cost- 
Effectiveness and the Weighted Scoring methods. 

Chapter V evaluates each of the five major activities in the Navy selection 
process in relation to the methods described in Chapter IV. Chapter VI summarizes 
the findings of the study. 
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Statem e nt of Resea rch Question 
This thesis provides answers to the following questions: 

o What is the relationship of the Navy's system to alternative systems that 
might be used? 

o What a re the significant: features of the competitive computer selection 
used by the United States Navy? 
o What alternative methods could be used? 
o What are the strong points in the Navy's system? 

o What are the controls imposed by higher authority? 

o Who in the Navy is responsible for the selection process? 



CHAPTER II 



THE GOVERNMENT ENVIRONMENT 



The system the United States Navy uses to competitively select computers is 
a result of the interaction of many and diverse influences. There arc strong- 
pressures exerted on the Navy by such external forces as the United States Congress, 
the President of the United States, the Bureau of the Budget, the Department of 
Defense and many others. In addition, there are active forces within the Navy 
itself. The most significant action element in the Navy is the Automatic Data 
Processing Equipment Selection Office. The purpose of this chapter will be to 
define these influences. The effect of these forces will be examined in Chapter ILL 

Congressional Influence 

The most significant influence exerted by the Congress is Public Law 89-306 
enacted on October 30, 1965. This law is known as the Brooks Bill because the 
legislation is a product of the Government Activities Subcommittee of the House 
Committee on Government Operations, Congressman Jack Brooks is chairman 
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of this subcommittee. ^ The need for this legislation was defined by the subcommittee 
in its report on II. R. 4845: 

Since 1958, up to the time of the hearings on H. R. 4845, the GAO has issued 
approximately 100 audit reports to agencies, congressional committees, and to 
Congress revealing serious shortcomings in the manner in which specific agencies 
acquired and/or utilized ADP equipment. The major deficiencies cited in these 
reports have been: 

(a) Inadequate feasibility studies; 

(b) Uneconomical and ineffective equipment utilization; 

(c) Overpayments resulting from inadequate management practices; and 

(d) Uneconomical procurement of equipment. ^ 

Based on this need, the Brooks Bill purports to satisfy three vital management 
needs: (1) more adequate management information, (2) optimum utilization of ADP 
(Automatic Data Processing) equipment through sharing, and (3) multiple use and 
economic acquisition. The subcommittee indicated that an economical acquisition 
program involved three principal factors: 

(a) Improving the Government’s bargaining position through volume acquisition; 

(b) Basing lease versus purchase evaluations, whenever possible, on the long- 
term value of the equipment to Government as a whole; and 

(c) Selecting that equipment for purchase which, on a Government-wide basis 
Offers the largest purchase advantage. 3 



■^U.S., Congress, House, Committee on Government Operations, Automati c 
Data Processing Equipmen t, Hearings, before a subcommittee of the Committee on 
Government Operations, House of Representatives, on H.R. 4845, 89th Cong. , 1st 
sess. , 1965, p. II. 

^H. R. Rep. No. 802, 89th Cong. , 1st sess. , 

^ lhid . , p. 23. 



1965, p. 17. 
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To satisfy the needs of management as cited above, Public Law 89-306 
increased the authority and responsibility of three federal agencies: the General 
Services Administration, the Bureau of the Budget, and the National Bureau of 
Standards in the Department of Commerce. 

The Bureau of the Budget is charged with the fiscal and policy control of ADP 
management of the government. The bill provides that the actions of any agency 
taken under the authority of the law arc subject to the review and approval of the 
Bureau of the Budget. 

The Administrator of the General Services Administration (GSA) is charged 
with the primary responsibility for coordinating the government's ADP management, 
subject to the policy and fiscal control of the Bureau of the Budget. The law 
specifically directs the administrator to "coordinate and provide for the economic 
and efficient purchase, lease, and maintenance of automatic data processing 
equipment by Federal agencies . ”4 The bill however docs limit this authority. Each 
agency was given the latitude to continue determining what their ADP requirements 
were. 

The National Bureau of Standards is responsible for the technical aspects of 
this coordinated management program. The Bureau is responsible for conducting 
research and development programs based on the needs of the government. The 



^Federal Property and Administrative Services Act of 1949 (63 Stat. 377) as 
amended, Sec. 111. (a). 
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drafters of the bill intended for the Bureau of Standards to direct its attention toward 
the problem of lack of equipment compatability. The intent of the law however was 
not to authorize the Bureau to launch a broad research and development program. 
Rather, the intent was aimed at supplementing the government research effort in 
coordination with other federal agencies and monitoring developments in the industry 
affecting areas covered by the law. A copy of Public Law 89-306 is shown in 
Appendix A. 



Presidential Influence 

President Lyndon B. Johnson on June 28, 1966, sent a memo to the heads of 
all federal agencies. The purpose of the memo was to stimulate action to improve 
the management of the ADP resources in the government. The President referred 
to the recognition that the Congress had given to this problem by enacting the Brooks 
Bill. President Johnson pointed to the fact that the federal government was using 
2,600 computers, employing 71,000 people in the field, and spending over $2 billion 
annually to acquire and operate ADP equipment. The President said: "I want the 
head of every Federal agency to explore and apply all possible means to use the 
electronic computer to do a better job . . . manage computer activity at the lowest 
possible cost. "5 The President pointed to four specific areas that were to be 

^Lyndon B. Johnson, Memorandum to Heads of All Federal Agencies, 28 June 

1966. 
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given priority attention: better methods of procuring ADP systems; achieving 
better utilization of existing capabilities through sharing and joint use arrangements; 
re-utilizing excess equipment when possible; and achieving greater computability. 

The President indicated to the Heads of the Agencies that he expected them to 
cooperate fully with the Bureau of the Budget, the General Services Administration, 
and the Department of Commerce in accomplishing these actions. (See Appendix B 
for a copy of the President’s memorandum.) 

Secretary of Defense Influence 

Secretary McNamara acted upon the President's memorandum with a statement 
of policy to the top management of the Department of Defense (DOD). ^ The Secretary 
indicated his desire to attain the objectives set down by the President but further a 
desire to set an example to die rest of the government agencies. The memorandum 
pointed to die fact that DOD operated 2,000 of the 2,600 computers cited in the 
President's memo and employed 51,000 of the 71,000 federal employees engaged in 
the ADP effort. It was therefore die view of the Secretary of Defense that DOD had 



Robert S. McNamara, Memorandum to The Secretaries of die Military 
Departments, The Chairman of the Joint Chiefs of Staff, The Director of Defense 
Research and Engineering, The Assistant Secretaries of Defense, The Director of 
Defense Agencies, 29 July 1966. 
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to bear a large share of responsibility for improved management of the governments 



ADP resources. 



The Secretary's memorandum defined the areas that required management 
attention. There are several items of specific interests in this paper: 

(a) We must, prior to computer selection, develop and issue systems 
specifications which adequately describe llie systems to be performed and which 
will result in selection of computers which can satisfy the requirements of that 
specification. We should not be forced to acquire additional units at later dates 
or replace the computers prematurely because of selections based on inadequate 
system specifications or on less than responsive vendor proposals. Military 
Dep artments and DOD agencies should centrally prescribe and control the 
development of systems specifications for computer acquisition purposes. 

(b) We must make competitive computer evaluation and selection a 
professional endeavor and organize and staff for it accordingly. Staffs performing 
this function for the senior automatic data processing policy officials should be 
divorced from computer- using organizations. Steps should be taken to assure 
that these staffs are the sole point of contact with the computer industry on all 
matters pertaining to computer selection for their respective agencies. 

(c) ... Further, computers will not be selected until the performance of the 
complete hardware/ software package required in the systems specification and 
request for proposal has been clearly demonstrated by either a full-scale or 
bench mark test . ( E mph a si s supplied . ) ^ 



Secretary of the Navy Action 



The Secretary of the Navy initiated a study effort within the Department of the 
Navy to define the actions that should be taken to satisfy the policy statements of 
President Johnson and Secretary McNamara. 



^Ibid. 
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An instruction signed by Secretary of the Navy Paul H. Nilze on June 30, 1967 

established the Automatic Data Processing Equipment Selection Office (ADPE SO). 

The mission assigned to this office is: 

a. To evaluate and select for approval by the Special Assistant to the 
Secretary of the Navy (SASN), automatic data processing equipment to be 
acquired by the Department of the Navy, except as defined in paragraph 4 
below. 



b. To act as the Department of the Navy contracting office for the 
procurement of ADPE. 

The exception mentioned above relates to equipments integrated with weapons 
systems, punch card equipment, and other special situations exempted by SASN. 

The functions assigned ADPESO are: • 

(a) Serve as the principal point of contact with the industry on matters 
concerning validation, evaluation, selection, and procurement of ADPE for the 
Department of the Navy. 

(b) Develop and administer for the Department of the Navy plans, policies, 
procedures and methods governing ADPE specification preparation, proposal 
solicitation, proposal evaluation and selection. 

(c) To solicit sources, negotiate and award contracts, place delivery orders 
against Federal Supply Schedules and administer contracts and orders for such 
ADPE acquisitions. 8 

The Director of the Automatic Data Processing Equipment Selection Office, 
reports to the Special Assistant to the Secretary of the Navy. Thus he is very high 



8 U.S. , D: apartment of the Navy, "Automatic Data Processing Equipment 
Selection Office; establishment of, " Secretary of the Navy Instruction 5430. 81, 
Department of the Navy, Washington, D. C. , 20390, June 30, 1967. 
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iii the Navy department hierarchy and is separated from computer- using organizations. 
(See Appendix D for a diagram of this organizational alignment.) 

ADPESO Action 

Upon completion of the first milestone of organizing and staffing, management 
turned and directed it attention toward the mission assigned to ADPESO. In order for 
this to be done, the methods of operation had to be defined and approved. This effort 
required ADPESO personnel to gather the various directives and regulations that 
apply to ADP management in the government. There is a large number of directives 
toward this end. (See Appendix E.) For the purposes of this paper however, only 
the most significant ones will be discussed. 

Bureau of the Budget Circular No. A- 54: 

Policies on Selection and Acquisition of Automatic Data Processing Equipment 

The purpose of this directive is to prescribe the government's policy 
concerning ADP equipment selection and provide guidance relative to the method of 
acquisition- lease, purchase or a lease with an option to purchase method. 

The government's policy concerning selection specifies: that all selections 
will be made based on a set of systems specifications; that selection procedures will 
assure equal opportunity for all manufacturers to participate; that the two prime 
factors to be considered in the selection process are (1) the equipment’s ability to 
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fulfill the systems specification, and (2) its overall costs including those applicable 
to acquisition, preparation for use and operation. 

The agency in question is obliged to select that method of ADPE acquisition 
which offers die greatest advantage to the government. The purchase method is to 
be used when all the following conditions exist: 

o There is a reasonable expectancy that the ADPE in question can be 
successfully and advantageously used; 
o That after a comparative cost analysis has been conducted, it is 

established that a cost advantage can be obtained in six years or less 
and the cost of money9 has to be used in conducting this cost analysis; 
o The ADPE will be able to satisfy the requirements of the system beyond 
the point where the purchase method provides a cost advantage; and 
o The feasibility and economics of performing equipment maintenance with 
in-house resources lias been considered. 

The lease-witli- option- to-purchase method is to be used when it is necessary 
or advantageous to acquire the equipment, but it is preferable to defer temporarily 
the decision to purchase when circumstances do not fully satisfy the conditions 
which would indicate purchase. 



9 

The procedure used is defined by Bureau of the Budget Circular A- 94. 
sample problem from this circular is shown as Appendix J. 



A 
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The lease method, without option to purchase, should be used when it is 
advantageous or necessary to acquire the equipment and when it has been determined 
that anyone of the prerequisite conditions of purchase cannot be attained. 

Department of Defense Directive 4105. 62: 

Proposal Evaluation and Source Selection 

The purpose of this directive is to establish the objectives, principles, and 
policy for the evaluation -of proposals and the selection of contractual sources. The 
directive defines three very significant parties and their roles in the selection 
process. 

First is the Source Selection Authority (SSA). He is an official designated to 
direct the source selection process and to make the source selection decision. The 
principal duties of this official are: 

1. Designate the chairmen and the membership of the Source Selection 
Advisory Council. 

2. Establish guidelines for the actions to be taken by the Source Selection 
Advisory Council and the Source Selection Evaluation Board. 

3. Make the decision as to the source, after an in-depth review and 
consideration of all information and data available from the Source Selection 



Advisory Council and the Source Selection Evaluation Board. 
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Second is the Source Selection Advisory Council (SSAC). It is composed of a 
group of senior military and civilian personnel and is appointed by the SSA and acts 
as his staff and advisors in the source selection process. The principal duties of 
this group are: 

1. Establish the evaluation criteria and assign weight factors to each item 
using the recommendations of the Selection Plan which was prepared by the System/ 
Project office, in this case ADPESO. 

2. Designate the chairman and members of the Source Selection Evaluation 
Board, 

3. Review the findings of the Source Selection Evaluation Board, 

Third is the Source Selection Evaluation Board (SSEB), This is a group of 
military and civilian personnel, from the various functional and technical areas. 

The duties of this group are: 

1. Evaluate and score the proposals submitted by the bidders; and 

2. Prepare a narrative justification for the evaluation results. 

This directive thus establishes a three-level review process for the evaluation 
of system proposals. 

Based on these various directives and the knowledge and expertise of the staff 
of ADPESO, the Navy’s computer selection process was developed. While the many 
directives provided broad policy guidelines, detail methods or techniques were not 



specified. 
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The method of evaluating the proposals is based on a system called the 
"Cost- Value Technique" from a book by Edward O. Joslin, Computer Selection . 10 
Mr. Joslin wrote this book while attending Boston College and working for EDP 
Equipment (Selection) Office of the U.S. Air Force. Currently he is Head of the 
Techniques Development and Analysis Division at ADPESO. 

The next chapter will treat the details of the Navy's Competitive Computer 
Selection System. 



^Edward O. Joslin, Computer Selection , (Reading, Massachusetts: Addison 
Wesley Publishing Company, Inc., 196S), p. 20. 
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CHAPTER III 



U.S. NAVY COMPETITIVE COMPUTER SELECTION SYSTEM 

The Navy’s competitive computer selection system starts when the system 
specifications for a particular data system are sent to the ADPESO for action. The 
actions taken by ADPESO can be arranged into five steps: 

Specification Review, 

Preparation of 'the Request for Proposal (RFP),and Selection Plan, 
Vendor Liaison and Proposal Validation, 

Proposal Evaluation, and 
Negotiation and Contract Award. 

Each of these areas will be discussed and explained. 

S pecification Review 

The ADPESO actually performs three different reviews of each set of 
specifications. The purpose of the first is to determine if the user's requirement 
can be satisfactorily satisfied by sharing a computer system currently operative in 
some other area, or by reutilization of some system not in use or scheduled to 
become idle and owned by the government. The second review is conducted to 
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determine whether this requirement should be satisfied by a competitive or a sole 
source procurement action. While these two reviews are important they arc not 
germane to the subject of this paper; accordingly attention will be devoted to the 
third review only. 

The third review performed by ADPESO is concerned with the data contained in 
the system specifications. 1 Three parts of the user's requirements are analyzed. 
First, the justifications provided for conditions stated as mandatory are checked. 
Second, the review examines the rationale provided for the requirements described 
as desirable features. The last part of this review is concerned with the workload 
description. 



Justification of Mandatory Requirements 

A mandatory requirement is defined as, "those requirements which the supplier 
must satisfy to be considered. "2 The Navy's policy is that the mandatory requirements 
shall not be stated in a manner that will restrict the free competition among all of 
the qualified bidders. Further the requirements cannot be oriented toward any 
specific vendor's equipment, software, or services. 



^U.S. , Department of the Navy, Specification, S election, a n d Acquisition o f 
Automatic Data Processing Equipment , Secretary of the Navy Instruction 10462. 13, 
Department of the Navy, Washington, D.C. 20390, p. B-2. The format for the 
systems specifications is shown in Appendix F. 

2 Ibid. , p. B- 11. 
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The specifications must he reviewed to insure that the user has justified 
all mandatory requirements, and that the number of them is held to a minimum. In 
addition the reviewer must insure that the requirement does not restrict free 
competition in any way. The justification for any mandatory item has to be expressed 
in terms of " . . . why alternative approaches would not prove satisfactory. This 
justification should show that the cost of effectively satisfying the requirement in 
any alternate way would be prohibitive, or that the required effectiveness can only 
be satisfied in the way shown. "3 A special check is also made to ensure that 
mandatory supplier capabilities, i.e., system maintenance, training, and conversion, 
etc., are not confused with mandatory system requirements. An example of some 
mandatory requirements and the types of justification required is shown in Figure 1. 

Following are examples of requirements that could, if properly justified, be 
mandatory. ^ 

o Equipment Availability 

If the equipment must be installed and operational by a specific calendar 
date to meet the requirement of the system(s), the user indicates the latest acceptable 
operational date. It should be noted that to specify a date is an exception to the 
normal practice. 



3 Ibid. , p . B- 2 . 

4 Il> i cl . , p. B- 11. 
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Mandatory Requirements 


Justification 


a. Supplier Support •• The supplier 
must agree to: 




(1) Train or provide training for, 
system analyst, programmers, 
and operating personnel. 


(1) The supplier must at least be 
responsible for the training to 
assure that the training is of the 
proper type. If this training does 
not provide the proper information 
the system cannot be properly 
utilized. 


(2) The maintenance personnel 
provided must have a SECRET 
Security Clearance. 


(2) This is necessary to meet local 
security requirements. See 
paragraph 


b. Workload Demonstration - The 
benchmarks must be successfully 
run in 5.00 hours or less. 


Completion of the benchmarks is 
required to demonstrate the timing- 
capability of the proposed system. 
Failure to complete the benchmarks 
in less than 5.00 hours would indicate 
that the supplier's equipment could not 
complete the activity’s projected work- 
load in the 600 hours a month available. 
Completion of the benchmarks will a 
also provide the validation of the 
existence of a suitable executive system 
and compiler. 


c. COBOL Compiler - The COBOL 

Compiler proposed and demonstrated 
with this system must be a USASI 
Standard COBOL Compiler (X3. 23- 
1968). 


This is required for compliance with 
Department of Defense acceptance of 
the USASI Standard. 


SOURCE: U.S. , Department of the Navy, Specification, Selection, and 
Acquisition of Automatic Data Processing Equipment, Secretary of the Navy 


Instruction 10462.13, Department of the Navy, Washington, D.C. 20390, p. B-12. 



Figure 1 — Mandatory Requirements and Their Justification 
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o Special Input/Output Requirements 

If special or nonstandard equipment must interface with the required 
equipment, or if some special characteristics are required, such as special 
character sets for the printer, minimum print characters per line, etc. , the 
requirement must be stated in such detail that the vendors will understand it. 
o Operational Requirements 

The user would define here any requirement that exists for a reserve 
for planned expansion, mobilization, and unavoidable workload peaking. Additionally, 
any requirement that all or certain portions of processing must be accomplished 
within a certain time frame are to be indicated. If the system is a real-time system 
the user can indicate the response time required. If the user has a peak load that 
must be completed during a specific time frame, the requirement must be stated 
in such sufficient detail that the suppliers can properly evaluate the situation, 
o Communications 

The user may have special communications requirements, such as 
remote stations, interface with communications network terminals or tributary 
terminal devices, etc. If this is the case then the requirements must be defined so 
the vendors can respond properly, 
o Security Requirement 



This is particularly applicable to Department of Defense activities. If 
the user's system does involve national security matters the rules applicable to the 
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supplier and the proposed installations must be stated. In addition the applicable 
regulation must be cited. 

o Expansion Capability 

The user should state the expansion capability required of the system for 
each year of the systems life. This is specified as a percentage increase over the 
first year, or by increased yearly extension factors. 

There are other mandatory conditions which might be specified. Normally the 
user will state the amount of time, per month, that is acceptable for completion of 
the workload. This is determined based on the time required to complete the 
benchmark. The determination process will be reviewed later. 

The experience to date indicates that most conditions claimed as mandatory 
are not usually sufficiently justified, so the requirement is classified as a desirable. 
It is also possible that as the threshold or limit changes, the condition can be 
treated as a mandatory requirement under one set of criteria and also as a 
desirable feature using a different criterion; for example, it might be a mandatory 
requirement that the supplier install the proposed equipment not later than a given 
date, and desirable for it to be installed earlier. ^ 



E.O. Joslin, "Competitive Computer Selection Within the Department of the 
Navy," (unpublished paper presented to the Diebold Croup, Inc., October, 1969. 
Mr. Joslin is Mead, Techniques Development and Analysis Division of ADPESO). 
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Rationale For Desirable Features 

A desirable feature is defined as "those features, characteristics, and 
services which the user desires, but which are not mandatory for a supplier to 
provide to be considered as meeting the overall requirement. "6 

For each desirable feature specified by the user, a statement of value lias to be 
given as a justification. The value has to be stated in terms of dollars over the life 
of the system. However, the rationale required to define a desirable condition is less 
rigid than that required of a mandatory item. A review of statements of desirable 
features, will attempt to find two characteristics. First: is the item adequately 
described and second does the rationale show why the item is desirable. The only 
acceptable explanation of desirability is one expressed in dollar terms. In fact when 
possible the user is suppose to provide two value or worth figures. One figure 
defines the savings or cost avoidance that will be realized if the feature is provided. 
The second figure provides the cost of achieving this desirable in any possible 
alternative way. The lower of these two figures is the value assigned to that feature. 
Following is a list of the type items which may be listed as desirable features. 7 
o Equipment A vailabil ity 

The user can indicate the desirability of having the equipment installed 
as much as three months in advance of the required delivery date. 

6 SecNavInst 10462. 13, p. B-ll. 



^ Ibid. , p. B-14. 
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o Software 

If the user desires any software packages to be provided by the vendor, 
each one is described. Some examples are: report generators, sort routines, 
scientific subroutines (such as general factor analyses, multi- discriminant analyses, 
etc.), and PERT cost and time, 
o Supplier Support 

The user can specify items of support desired of the supplier such as, 
training to be conducted on-site, some number of man-months of analyst support in 
conversion of programs to the new system, and on-site maintenance, 

o Mathematical Computation 

Special mathematical capability can be requested such as the ability to 
handle floating point computation, 
o Utilization Log 

The user can request that the equipment, in conjunction with software, 
provide a daily utilization report by job number. 

Figure 2 shows some examples of desirable features and the corresponding 
value statements. 



Statement of Workload 

During the review of the mandatory and desirable features, the 
reviewer must also concern himself with the workload definition. This is so 
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Desirable Features 


Value Statement 


1. Flow charting 
program for 
COBOL 


It is estimated that a potential cost avoidance of $60,000 
would accrue to the government with the use of a Flow 
chart preparation program. Tins is based upon 
programmers devoting 5 per cent of their time to the 
task of flow charting. A staff of thirty earning an 
average of $9,000 annually for four years is $1,080,000. 
Five per cent of this figure is $54,000. However, it is 
estimated that this would cost 528 hours of computer 
time over four years. At $37. 50 per hour (non-peak 
time value), this is $19,800 total cost. Therefore, a net 
savings of $34, 200 will result. An alternate source of 
supply for such a program indicates a cost of $8,000. 
This cost was obtained from quotes received from 
private industry suppliers. Therefore, this alternate 
source cost of $8,000 is taken to be the value of this 
program. 


2. Expansion 

capability of up to 
fifty per cent over 
the stated life 
workload 


A system which would require the full five hours to meet 
the mandatory requirement for completing the 
benchmarks would have the required life of five years 
and is expected to cost $1,000,000. However, the stated 
system life is based on expansion of the workload which 
has already been assigned to this organization. In 
addition to the capability to handle the known workload, 
it would be desirable to be able to handle two additional 
projects which have a fifty per cent chance of being 
assigned to this organization. The first project (project 
XYZ) would require an additional capability of approxi- 
mately thirty per cent of that required for the stated 
workload. The second project (project ABC) would 
require an additional time capability of twenty per cent 
and would also require the availability of a remote tern 
terminal. Our ability to handle project XYZ could save 
an ultimate expenditure of $300,000 over a three year 
period in contracted computer time. Likewise, the 
ability to handle project ABC could result in savings of 



Figure 2 — Statements of Value 
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Desirable Features 


Value Statement 




$200,000. However, because there is only a fifty per 
cent chance of these projects being assigned to this 
organization only half of the possible $500,000 saving, or! 
$250,000 is being stated as the value of this expansion 
capability. 


3. Supplier provided 
on-site training of 
sixty 

programmers, 
thirty analysts, 
and ten operators 


If this training is not provided by the supplier, a cost of 
approximately $20,000 would be incurred in hiring a 
consultant to supply this training in accordance with the 
supplier’s direction. If this training is not accomplished 
on-site, an additional cost of approximately 200 man- 
weeks (assuming two week courses) of per diem or 
$19,200 plus travel cost could be incurred. 


4. On-site 

maintenance 


If on-call maintenance is all that is offered, this would 
mean one or two hours of additional lost time per call, 
and with the expected five calls a month, this could mean 
ten hours of lost time per month or 600 hours over the 
life of the system. This lost time is estimated to be 
worth $12, 000. 


SOURCE: U.S. , Department of the Navy, Specification, Selection, and 
Acquisition of Automatic Data Processing Equipment, Secretary of the Navy 


Instruction 10462.13, Department of the Navy, Washington, D.C. 2-390, p. B-15. 



Figure 2 — — Statements of Value - Continued 



because many of the mandatory and desirable features relate to the vendor's 
capability to complete the workload satisfactorily. 

The review of the using activities workload is concerned with insuring that the 
expected workload is described in terms of the expected levels of workload and in 
terms of the representative programs or tasks plus the related extension factors 
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applicable to the workload levels. Each of these factors, workload levels and 
representative programs, will be discussed individually. 

Expe cted Workload Levels 

The workload to be processed on any system can be considered as consisting of 
a series of various levels. As the system ages, the workload usually increases 
from one level to the next. However, because of the problems associated with 
projecting future workload growth, it would be unreasonable to expect that the user 
could be 100 per cent accurate in projecting die workload for the life of the system. 
Accordingly the user is directed to use probabilities to predict future workloads in 

g 

the following manner: 

1. System Life - the user prepares a chart that shows the number of years 
the system will be in existence. 




^Joslin, 



"Competitive Computer Selection,” p. 



13. 



Workload Workloa' 
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2. Projected Growth - an approximation of the predicted growth in 
workload is superimposed on the basic "System Life" chart a. . The vertical axis 
represents workload in hours-per -month. 



b. PROJECTED GROWTH 



T5 



1st Year 




3. Workload Levels - the user constructs a workload level line by drawing 
a horizontal line through the midpoint of the projected growth line for each year. 



c. WORKLOAD LEVELS 













Level 5 
Level 4 
Level 3 
Level 2 
Level 1 
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1st Year 


2nd Year 


3rd Year 


4th Year 


5th Year 



4. Level of Probability - for each year a probability factor is assigned to 
each workload level. The total probability for any one year is considered to be 100. 
In illustration d. the user has indicated that during die first year there is a 90 per 
cent chance that the workload will be at Level 1 and a 10 per cent chance it will be at 



Workload 
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Level 2. Note in the illustration below that the number of workload levels was 
increased to seven. This was done to take care of a situation where there is a 
probability the workload will exceed the projected growth line in the fourth and fifth 
years. 



d. LEVEL OF PROBABILITY 



0 


0 


0 


0 


5 


0 


0 


0 


5 


15 


0 


0 


0 


15 


70 


0 


0 


10 


75 


10 


0 


10 


80 


5 


0 


10 


85.' 


5 


0 


0 


90 


5 


0 


0 


0 












1st Year 


2nd Year 


3rd Year 


4th Year 


5th Year 



Level 7 
Level 6 
Level 5 
Level 4 
Level 3 
Level 2 
Level 1 



Representative Programs 

This is a very important part of the systems specifications. The representative 
programs are the vehicles that will be used to time the proposed computer systems. 
The essence of this approach is that a program called a "representative program" is 
selected to represent a specific portion of the total monthly workload. Thus, a group 
of representative programs represents the total monthly workload. These programs 
which become benchmarks, in conjunction with the extension factors, provide the tools 
to project the monthly running time for the systems proposed by each vendor. 

In view of the importance of this factor the review conducted by ADPESO is 
particularly critical. The reviewer looks to insure that certain facets are properly 
addressed. For example, the benchmarks should be written in a standard higher 
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level programming language such as: USASI Fortran or USASI COBOL. The vendor 
should be able to run all programs during a single half-day benchmark demonstration. 

The reviewer lias to insure that the user has selected a truly representative 
program. This means that the benchmark problem must be representative of the 
following:^ 

1. Types of processing: logic, computation, housekeeping, etc. 

2. Time requirements: compile, execute, tapes, card units, printers, etc. 

3. Equipment Requirements: core, input/output channels, tapes, card units, 
printers, etc. 

4. Order of problems (priority). 

The review also has to establish that the user has properly equated each 
benchmark to a portion of the workload. This is particularly important for as 
previously mentioned, the performance demonstrated on the benchmark is extra.- 
polated to define the total monthly workload. 

The extension factor, a number derived for each representative program, is 
the next vital element to be verified. The running time of each benchmark is 
multiplied by this factor to arrive at the monthly running time for the class of tasks 
represented by that program. An extension is required for both sequential systems 
and for multi-processing systems. The user should have prepared a table similar 
to Figure 3. For each representative program there should be entries in the table 
showing the total monthly time consumed by each program, the running time of the 
representative task or program and the division of the total by the representative 



^ Edward O. Joslin, Computer Selection (Reading, Massachusetts: Addison- 
Wesley Publishing Company, 1968), p. 77. 
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Workload 


(Hours r 


Time 

md Hundredths) 












Extension 










Task Set 


Functions 


Monthly 

Task 


Representative 

Task 


Factors 






(Single Run) 




Sort 


Total thruput 


145.00 


0.45 


322 


B-1A 


Mag. tape 


125.00 


0.25 


500 




Card Reader 


115.00 


0.03 


3833 


Edit 


Total thruput 


120.00 


0.75 


I GO 


E-4a 


Mag. tape 


80.00 


0.G0 


1 O O 

J.O U 




Card Reader 


20.00 


0.50 


40 




Printer 


100.00 


0.25 


400 


Update 


Total thruput' 


100.00 


0.16 


625 


D-5a 


Mag. tape 


70.00 


0.10 


'700 




Card -Reader 


25.00 


0.05 


500 




Printer 


50.00 


0.10 


500 


Matrix 


Total thruput 


90.00 


0.45 


200 


Inversion 


Card Reader 


3.50 ■ 


0.02 


175 


K-Ga 


Mag. drum 


24.00 


0.15 


160 




Printer 


1.50 


0.01 


150 


FORTRAN 


Total thruput 


85.00 


0.17 


500 


Compile 


Mag. tape 


78.00 


0.15 


520 


II- 3 a 


Card Reader 


6.00 


0.02 


300 




Printer 


4.00 


0.01 


400 


COBOL 


Total thruput 


40.00 


0.12 


333 


Compile 


Mag. tape 


38.00 


0.11 


345 


.G-2 ' 


Card Reader 


4.00 


0.04 


100 




Printer 


3.00 


0.04 


75 


Tape to 


Total thruput 


300.00 


1.00 


300 


Print 


Mag. tape 


300.00 


1.00 


300 


P-4 


Printer 


300.00 


1.00 


300 




TOTAL MONTHLY 








TIME 


. . 880.00 







SOURCE: U.S. , Department of the Navy, Spe cification, Selection, an d 
Acquisition of Automatic D at a Processing E quipment , Secretary of the Navy 
Instruction 10462. 13,— D epartment of the Navy, Washington, D.C. 2 0390, p . B-4. 



Figure 3 — Representative Programs 
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program and this becomes the extension factor. Expressed another way, the 
extension factor is: 



X 


Y 

Z 


Y ■ 


= Total monthly time to perform total task set on 

present equipment system 


Z 


= Throughput time to run the representative 

program 

(1) on present equipment (subscript a) 

(2) on proposed equipment (subscript b) 


'' X 

Therefore: 


= Extension factor for representative task. 


X(Zb> 


~ Suppliers time for total task 10 



The user also has to show how these individual representative programs can 
be grouped into a mix of programs that has its own extension factor, which is 
representative of the total workload. The purpose of this mix of representative 
programs and related factors is for use in demonstrating the multiprogramming 
or multiprocessing capability. The mix of programs as modified by quantity and 
provisions, should be such that when the times for the present representative 
programs are multiplied by the extension factors the result will be the total time 
which should closely approximate the present total time. Figure 4 demonstrates 
this situation. 



10 



SecNavInst 10462. 13, p. B-5. 
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Program 


Quantify 


Provisions 


B-la 


2 


Normal 


E-4a 


1 


Input from tape 


D -5a 


4 


Twice Normal 
Twice no output 


K-Ga 


1 


Normal 


11- 3 a 


3 


Normal 


G-2 


2 


Normal 


. F-4 


2 


Normal 


Extension Factor for Mix: 160* 



"This extension factor for the mix is derived by examining the information contained 
in Figure 4 and obtaining the lowest practical extension factor to reduce the number 
of problems to be run in the mix while retaining the required representative nature cf 
the mix of problems, which in this case is 160. This extension factor is then 
divided into each of the sequential extension factors to obtain the quantity column. 

The provisions column is then used to make the input/output total time when extended 
by the mix extension factor equal to the total projected input/output time. This mix 
of tasks (15) can then be used as a proper demonstration of a supplier’s multi- 
programming processing . 

SOURCE: U.S. , Department of the Navy, Specification, Selection, and 
Acquisition of Automatic Data Processing Equipment , Secretary of the Navy 
Instruction 104-62. 1 3, Department of the Navy, Washington, D. C. 2 -390, p. B-6. 



Figure 4 — Extension Factors 



The extension factors initially calculated are for the first year of the systems 
life. Accordingly, the user must also provide extension factors applicable to the 
second, third, fourth, and fifth year of the systems life. 

The final check the reviewer makes is to insure that the user has provided a 
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description of each of the benchmark programs and copies of the program detail 
listing. 



If the user has properly justified the mandatory requirements, evaluated the 
desirable features, and described the workload, the specifications are ready for the 
next action. The specifications are forwarded to the Source Sek+fcrun Evaluation 
Board for the preparation of the Request for Proposal which will be forwarded to the 
vendors, and the Selection Plan which is held in-house for official use only. 



Preparation of the RFP and Selection Plan 

Provided the specifications were properly prepared, the preparation of the RFP 
and Selection Plan is a relatively simple task. ^ 

The RFP consists of a cover letter, the systems specifications, less the 
justification of mandatory requirements and the rationale for desirable features, and 
a set of instructions addressed to the vendors. These instructions tell the vendors 
how to prepare their proposals; lists some mandatory administrative detail. s and 
defines how the cost for the proposals should be computed and shown. Before tills 
can be mailed to the vendors through, tine Selection Plan has to be developed. 

The purpose of the Selection Plan is to establish a detail set of instructions that 
will govern how the vendor proposals will be evaluated. It should be noted at this 
point that in the Navy's system it is the desirable features requested by the user that 

* joslin, "Competitive Computer Selection, p. 22. 
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are evaluated. The mandatory requirements are not evaluated they are validated. ^ 

The desirable features arid the values assigned to each are carefully analyzed. 

Based on this information a value templet is constructed for each feature. These 

templets show how the total value for each item will be apportioned as various 

quantities or qualities of the item are proposed. In actual practice these value 

statements are prepared when the RFP is being prepared. The purpose of this is to 

insure agreement between the RFP and the Selection Plau. 

Another examination of the mandatory requirements is also conducted at this 

time. The reason for this second review of these requirements is to determine if 

any items initially classified as mandatory can be re- classified as desirable. 

Experience has shown that some mandatory items are really desirable in the sense 

that dollars can sometimes make up for a slight deficiency in a mandatory 
13 

requirement. J 

There are four parts in a value statement: 

1. A Statement of the Desirable Feature - provides the exact and entire 
statement of the desirable feature as it appears in the RFP. 

2. Rationale - This section shows the derivation of the worth or value of 
the feature. The worth is defined by considering the following: 

a. The cost of doing without the capability; 

b. The cost of satisfying the requirement by using in-house resources; 

12 SecNavJnst 10462.13, p. B-ll. 

^Joslin, "Competitive Computer Selection,” p. 23. 
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c. The cost of having the capability provided by some contractor other 
than the one bidding on the system (must always be preceded by a. or b. ); 

d. A cost less than any of the above which would reflect what the 
activity would apy to get this capability (must be preceded by a. , b. , or c.); and 

e„ The worth of each feature must be calculated for each year of the 
systems life. 

The lowest worth derived above becomes the value of the feature. 

3. Templet - tells how the worth or value derived is to be distributed. 

4. Valuation - is actually done on a separate standard form which is 
attached to the value statement sheet. (A sample value statement is shown in 
Figure 5.) 

The Selection Plan also defines how the costing of the vendors proposal will be 
accomplished. It is at this point that many of the previously mentioned elements 
start coming together, e.g. , the workload levels, extension factors, time required 
to execute a benchmark, etc. The cost of money is introduced at this point. 

There is a requirement to perform an economic analysis of the cost. ^ Currently 
the discount rate in use is 10 per cent. 

An example of the formula used to define the cost of a system is shown 
below. The basis for this example is the workload levels and probabilities 



■*^U.S. , Department of the Navy, "Economic Analysis of Proposed Department 
of the Navy Investments," Comptroller of the Navy Instruction 7000. 28 ( April 19, 
1967) Department of the Navy, Washington, D.C. 20390. 
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VALUE STATEMENT 



DESIRABLE FEAT URE : Capability to expand at year 2, the systems at the 

three activities to include time sharing systems for program testing 
purposes. 

RAT IONALE : "Time Sharing Versus Batch Processing: The Experimental 

Evidence" by H. Sockman, 10 October 1967, AD 661-665, Defense Documen- 
tation Center publication, reports that a 25% reduction in program 
development time has been shown in experimental studies by using time- 
sharing techniques instead of batch processing techniques in testing. 
Time-sharing caused a 40% increase in computer test time. No signi- 
ficant differences were found in program size or run time. On this 
basis, the follov/ing cost reduction would be gained from time-sharing: 

25% of 'all programmers 1 yearly salary - 40% X hourly 
computer rental rate X hours test time per week X 52 
weeks per year = cost reduct i on/base. 

Taking number of programmers at each branch - 20 

Programmer yearly salary = $20,000 

Hourly computer rental rate = $100 

Average hours test time per week = 20 

Then (.25 X $20,000 X 20) - (.40 X $100 X 20 X 52) = cost reduction per 
year for one activity. 

$100,000 - $41,600 = $58,400 yearly savings per activity 

X3 activities 

$175,200 total per year 

X4 years 

$700,800 total savings 

TE MPLET : If proposed, a value of $58,400 would be given for each activity 

per year for each year after the first year. 

SOURCE: E.O. Joslin, "Competitive Computer Selection Within the 
Department of the Navy," unpublished paper presented to the Diebold Group Inc. , 
October 1969. 



Figure 5 — Sample Value Statement 
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shown on page 28. In this case, there is a 90 per cent chance of the workload 
being at Level 1 for the first twelve months, and there is a 5 per cent chance this 
will be the case in the second twelve months. This same type of analysis is 
performed for all of the other workload levels. Based on this data tiie costing 
formula is then developed. The formula for the first world oad level would be: 
MC wl n = monthly cost workload level n 

Y = number of months 



P 

D 



probability of operating at ^ 
discount rate 



YMC wl n • P • D + YMC wl n • P • D 

12MC wl x x 0.90 x 0.91 + 12MC wl j x 0.05 = 10. 33IvlC wl 1 

By performing the same mathematical operation to the other workload levels, the 
present worth of total expected lease payments for each configuration can be 
expressed as: 

10.33MC wl x -h 10.00MC w12 + 8.56MC wl 3 + 7.75MC wl4 + 6. 89MC wl 5 + 
1-89MC w16 + 0.37MC w17 .‘5 

The Selection Plan is now complete. The SSEB presents the RFP and the plan 
to the Source Selection Advisory Council for approval. When this review is completed 
and approval is granted, the RFP is forwarded to the suppliers. 



^U.S. , Department of Defense Computer Institute,. Re port of the Fourth 
Computer Selection Techniques Seminar (Washington, D. C. : February 18- 19, 1969), 

p. 8. 
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Ve ndor L i aison and Validation 

The amount of effort that has to be devoted to this function is inversely related 
to the degree of excellence in the RFP. The period of time is normally four months. 
Essentially all that is required of the Navy personnel is to be available to answer 
vendor questions, and to be ready to witness the running of the benchmark programs 
when the vendors are ready to demonstrate them. 

There is one cardinal rule in vendor liaison: whatever is told or applies to 
one vendor must be told to all vendors, and in writing. This rule is considered 
essential - --* - it is often difficult to differentiate between liaison and negotiation. It 
is also necessary in order to avoid later difficulties which can arise when all 
bidders are not equally informed. 

Proposal Evaluation 

Evaluation of the vendors proposals is not a simple task if, however, the RET 
and Selection Plan is properly prepared the degree of difficulty will be reduced 
considerably. The actual steps included in the evaluation process are: 16 

1. Reading and understanding all proposals. 

2. Verifying that all mandatory conditions are satisfied. If a proposal does not 
satisfy all manditory conditions it is ruled not responsive and it is no longer 
considered. 

3. Determining the degree to which the requested desirable features have been 
provided. 



^Joslin, "Competitive Computer Selection, " p. 29. 
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4. Matching ihc response to the desirable features with the value statements. 

5. Costing the vendors proposal. 

The verification of vendor claims is often difficult. To assist in this, the RFP 
requests the vendors to reference the technical manuals. If the SSEB cannot verify 
a claim by using the technical manuals or other references, the capability in question 
is regarded as a vendor’s promise and it is incorporated as a term and condition in 

i n 

any resulting contract. 3 

Determining the degree to which a desirable feature is satisfied and matching 
this with the value statement is also a difficult task. "It can result in many bruised 
egos, but normally it can be worked out to the satisfaction of all. ”^ 8 

The pricing or costing of the life of a system is an involved process. The total 
cost is dependent on many elements: the number of years the system will be used, 
the number of hours the equipment will be used, the cost of the desh'able features 
provided by the vendor, the cost of satisfying a desirable feature by an alternate 
means if the vendor does not provide the item, the cost of vendor support, etc. 

Another complicating factor is that each of these costs have to be considered 
under several different procurement possibilities: lease, lease with purchase option, 
purchase, and any special procurement plan which may be worked out with vendors.-* 9 
In addition and as previously mentioned, an economic analysis using the present value 
technique must also be included in the calculations. ^0 



17 Ibid. 


18 Ibid. 


19 Ibid. , p. 31. 


^NavComptlust 7000.28 
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Negotiation and Contract Award 

The SSEB review of the vendor proposals, is attended by a continuous search 
for items which might be negotiable. An example is an item offered by one vendor 
at no cost might be obtainable free from the other vendors if the item was subjected 
to negotiation. Usually such items as training, maintenance, and some software 
packages fall into this category. 

After the SSEB has defined which proposals are responsive to the requirements 
of the RFP, letters are prepared and sent to the vendors. The purpose of this letter 
is to set up the time and place for negotiation sessions with the responsive bidders. 

In addition the letter tells the vendor what items the Navy wishes to negotiate with 
the vendors. 

At the negotiation session, the vendors and Navy personnel discuss the items 
in question. The vendor is told when to submit the final proposals. These proposals, 
however, must be based on the original proposal plus information covered in the 
negotiation sessions. Each of the contractors is encouraged to improve his proposal, 
and they are reminded to submit them by the scheduled date because late proposals 
might not be considered. The session is then closed, and a memorandum of the 
meeting is prepared and becomes a part of the contract pre- award file. ^1 

When the proposal changes are received, the SSEB again reviews the proposals 



2 



L Joslin, "Competitive Computer Selection," p. 32. 
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and adjusts the cost data based on the changes received. The evaluation then 
determines the proposal representing the lowest overall cost to the Department 
of the Navy. The selected proposal is nominated to the SSAC and to the Special 
Assistant to the Secretary of the Navy (SASN). 

When the SASN approves the selection, three copies of the contract are sent to 
the vendor for signature. If the supplier disagrees with some aspect of the contract, 
the point is discussed and resolved. The vendor's representative and the government 
contracting officer then sign the contract. 

Occasionally there is one other step. If the award is for more than $ 1,000,000, 
ADPESO does not announce the award for at least twenty working hours after notifying 
the Office of Legislative Affairs. The purpose of this hold is to allow the appropriate 
Congressman to be told of the award. ^2 

There arc several other actions that are initiated at this time. An announce- 
ment of the award is sent to the Commerce Business Daily for public announcement 
via that media. Letters are sent to the unsuccessful bidders indicating that their 
proposal was not selected however their future interest is solicited. This letter also 
provides the unsuccessful bidder an opportunity for a post- approval meeting with 
ADPESO. If any unsuccessful bidder desires such a meeting a written request has to 
be sent to ADPESO within ten days. There are several guidelines established 



22 



Ibid. , p. 33. 
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governing the conduct of these post- approval meetings: 

1. No informal sessions with respect to approved equipment selection shall 
be held with any unsuccessful supplier, nor shall any oral information be 
furnished on the subject. 

2. All questions from an unsuccessful supplier shall be provided to ADPESO 
in advance and in writing, and should be as specific as possible. The DON 
reserves the right to reject irrelevant or immaterial questions. Conversely, 
related and relevant questions which arc obviously and naturally evoked by 
responses to previously submitted questions may be entertained within reason. 
In this regard, however, the normal provisions apply with respect to security 
sensitive information and proprietary information. 

3. The judgement x exercised in the application of selection criteria to 
particular proposals is properly a matter wholly within DON purview. 

4. Post- approval meetings shall be held as soon after they are requested as 
practicable. ^3 
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SecNavInst 10462.13, p. Ill- 9. 



CHATTER IV 



ALTERNATIVE SELECTION SYSTEMS 



To assist prospective users, today there are a number of different methods 
used in determining which computer system best satisfies the data processing- 
requirements of the organization. Some organizations select their computers by 
soliciting competitive bids while others select a system without seeking any vendor 
competition. When competitive bids are requested the criteria used to determine 
the best system ranges from objective standards to subjective judgement. The 
purpose of this chapter is to review the various current practices and explore the 
details of some of the systems used. 

Survey of Computer Selection Practices 

Dr. Norman F. Schneidewind of the Advanced Systems Division, Systems 
Development Corporation, conducted a survey for Datamation and the results were 
subsequently published in the magazine. 1 The data was obtained by sending 



^Norman F. Schneidewind, "The Practice of Computer Selection," Datamation , 
13. 2 (February 1967), pp. 22-25. 
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questionnaires to four types of computer users. The types of users and the number 
responses from each were: 



1. Commercial: Non -government, non-aerospace, non-university 37 

2. Government: Civilian, military, federal, state, local 20 

3. Aerospace: Aircraft, missle, space 8 

4. University 4 

Total responses 69 



The survey questionnaire was directed toward four aspects of the computer 
selection process. The areas covered and the findings of the survey are as follows. 



Competition 



The competition section of the questionnaire was designed to determine whether 
the selection process takes place on a competitive or sole source procurement basis. 
The responses indicated that between 77 and 92 per cent of all users utilize 



competitive selection systems. The degree to which the various users rely on the 

O 

competitive process is shown below. 



Government 
Non-government 
Commercial 
Aerospace 
University 
Total All Users 



80.6% 

85.7% 

75.0% 



94.4% 

80.9% 



84.6% 



Dr. Schncidwind determined that in general it was the user with a small installation 



(average monthly rental of $16,800) that used single source procurement, while the 



2 Ibid. 
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management of larger installations (average monthly rental of $62, 300) select their 
equipment based on a competitive bidding system. 



Methods 



The purpose of the methods section of the questionnaire was to determine which 
of the following selection methods are used. The methods are listed with the one 
considered by Dr. Schneidewind as the most objective first while the last was 
considered to be the least objective: 

. Computer Simulation 
. Mathematical Modeling 
. Program and Execute Test Problems 



. Evaluation of Benchmark Problems 
. Published Hardware and Software Evaluation Reports 



It was found that larger installations use more objective methods than small 

3 

installations. The specific findings are given below. 





0 


$15,000 


$50,000 




to 


to 


and 




$15,000 


$50,000 


Up 


Benchmark Problems 


50.0% 


81.8% 


76.9% 


Evaluation Reports 


80.8 


77.3 


53.8 


Test Problems 


53.8 


54.5 


61.5 


Simulation 


7.7 


13.6 


46. 2 


Modeling 


0 


4.5 


30.8 



3 Ibid. 
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Selection Criteria 



The respondents were asked to rank the following computer selection criteria 
in order of importance: 



The user was asked to rank these items by assigning the number one to the most 

important, the number two to the next most important, etc. According to 

Dr. Schneidewind the most objective criteria aye hardware and software performance, 

while the lea st objective are ava ilabil ity of application programs and manufacturer support . 

The results indicate that all users place more emphasis on the objective 

(hardware and software) criteria than on the subjective criteria. Dr. Schneidewind 

considered this to be the most significant finding of the survey. In addition the results 

indicate that commercial users rated cost as a less important criteria than did the 

4 

other users. T he average rankings are shown below. 



H a rd \va re p erf or man ce 

Software performance 

Support provided by manufacturer 

Availability of application programs 

Compatibility with present hardware and software 

Delivery date 

Potential for growth (modularity) 

Cost 



Commercial Government Aeros pace Univer sity 



Hardware 

Software 

Manufacturer Support 

Availability of Programs 

Comparability 

Delivery 

Growth 

Cost 



3.11 2.15 1.57 2.50 

2.73 2.45 3.68 1.75 

3.92 4.32 5.29 3.50 

6.57 7.30 7.14 6.75 

4.35 4.85 3.86 6.00 

6.27 6.45 6.43 7.25 

4.68 4.25 5.57 4.50 

4.38 4.25 2.43 3.75 
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Outside Assistance 

The intent of this section of the questionnaire was to determine the extent to 
which users depend on outside assistance in the selection process and when they did 
what type of consultant was used. Specifically the respondents were asked to indicate 
which of the following types of outside assistance were used: 

Independent Consultant 
Accounting Firm 
Management Consultant 
Technical Firm 
Computer Manufacturer 

The findings indicate that the number organizations using outside assistance is 
small, arid the organizations that do use consultants generally have small data 
processing installations. When outside assistance is used the computer manufacturers 
are relied on more heavily than die others. 

Based on the analysis of die data accumulated hi this survey. Dr. Schneidewind 
developed what he considered to be a typical computer selection process. In general, 
computer selection is competitive, objective criteria is stressed, and the selection is 
performed in-house rather than by an outside consultant. 



Rev iew of Specif ic Alternative Systems 



The process of selecting a computer from among several alternative proposals 
logically breaks down into two major parts. The first part consists of tiie validation 
of the vendor's claims. The validation process is concerned with verifying the 
capabilities of each system proposed and the estimated system times required by 
each configuration to complete the workload. This is a very important part of the 
selection process. The user must be aware of the old saying "buyer beware." While 
the vendors will probably not perpetrate a falsehood the buyer must understand that 
the proposal is a selling device and certainly each vendor wants to look better than 
all others. In order to look best, the vendors will interpret the specifications in a 
way that is favorable to their equipment. The amount of interpretation is related to 
the way the requirements are set forth in the specifications of the RFP. If the 
specifications arc loosely worded then the suppliers are put into a situation that 
requires a great deal interpretation of the specifications to make a proposal. An 
example of the types of problems that can arise if the proposals are not thoroughly 
validated against the organizations requirements is the situation at United Airlines 
and Trans World Airlines: 

Two of the largest commercial computer systems ever ordered — for 
reservations at United Airlines and Trans World Airlines — are not working out. 
United has decided to drop its Univac system, estimated to be valued at more than 
$30 million, while TWA reportedly is close to canceling an equally large system 
ordered from Burroughs Corp. According to United, the Univac system was more 
than a year late and still inoperative. Industry sources report that delivery 
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problems in hardware and misunderstandings over programming tripped up 
Burroughs and TWA. Both airlines will continue to use their existing computers 
until they select suppliers for a third- generation system. 5 

The second half of the selection process is the evaluation of the proposals that 
satisfy all of the basic data processing requirements. The purpose of the evaluation 
is to differentiate among the various proposals and find the one that best satisfies all 
the selection criteria. 

There are several methods, besides the ones used by the U.S. Navy, that can 
be used in the validation and evaluation of system proposals. The rest of this chapter 
purports to examine some of the alternative validation and evaluation systems. 

Alternative Validation Methods 

Two areas of each proposal must be validated: system capabilities and system 
timing. 



Capabilities 

The user can verify system capabilities in several ways. First, all the 
manufacturers technical literature can be read. The next step is to compare this 
data with the vendors proposal. An alternate way would prepare a detailed 



^"Airline Computer Problems," Business Week , February 7, 1970, p. 34. 
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questionnaire to be included in the RFP and request the suppliers to provide the 
requested data. The vendors proposal is then compared with this data. A third way 
of obtaining this information is to study the Standard EDP Reports prepared by the 
Auerbach Corporation. ^ 

Timing 

The validation of system timing is a most important task. In addition, it is a 
difficult: task to accomplish, particularly with a high degree of accuracy. The 
importance of this operation however, can not be overemphasized. The timing- 
estimates derived will serve at least two vital functions in the selection process. 

First this information will be used to determine if the proposed configuration can 
perform the stated workload. Second, the timing data will provide the foundation for 
costing the proposal. 

There are several ways to verify die timing of computer systems: benchmarks, 
computer simulation, and detail hand timing. The use of benchmarks was described 
in Chapter III. 

Simulation 

Simulation is a technique in which a model of the actual system is described and 

constructed and testing of the real system is accomplished by exercising the model. 

^ Edward O. Joslin, Compu ter Selection (Reading, Massachusetts: Addison- 
Wesley Publishing Company, Inc. , 1968), p. 67. 
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The function then of a simulation model is to show the results created when certain 

7 

things happen under the control of a set of decision rules. The limitations and the 
advantages of simulation are stated by Teichroew and Lubin: 

Simulation problems are characterized by being mathematically intractable and 
having resisted solution by analytic methods. The problems usually involve many 
variables, many parameters, functions which are not well behaved mathematically 
and random variables. Thus, simulation is a technique of last resort. Yet, much 
effort is now devoted to "computer simulation" because it is a technique that gives 
answers in spite of its difficulties, costs and time required. 8 

There are several computer- simulation techniques in use or under development; 
however, the system most widely known and used is one called SCERT (Systems and 
Computers Evaluation Review Technique). ^ SCERT has the capability to simulate 
over 100 different computer systems manufactured by the thirteen different companies 
companies. A complete list is shown in Appendix G. SCERT divides into four 
major components: definition languages, a factor library, simulation programs, 
and output repoi'ts. ^ 

The definition languages are used to define the applications systems to be 
simulated and the hardware/software complexes to be simulated. Several 



^John E. Cremcans, "The Trend in Simulation," Computers and A utomation , 
January, 1968, p. 46. 

g 

Daniel Teichrocu and John F. Lubin, "Computer Simulation-Decision of the 
Technique and Comparison of Languages," Communicat ions o f the ACM , IX (October, 
1966), p. 724. 

^Joslin, Computer Selection, p. 86. 

^ A Te chnical Descr ip tion of SCE RT (Rockville, Maryland: Comress, Inc.), 
pp. 17-49. ' 
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language section components facilitate this. The data definition division defines 
the files and reports of the applications system. The procedures division defines 
each application program. The configuration division defines each equipment 
configuration to he simulated, including model number and quantity. The environment 
division defines the staff that support the installation plus salary level. 

The factor library contains die characteristics of the hardware and software 
items — characteristics such as cost, performance and technical specifications. This 
data is obtained by Comress from the manufacturers. 

The simulation programs perform the processing necessary to accept the input 
definition data and create the output reports. ‘ The programs are used in five phases 
which will be discussed later. 

The output reports consist of fifteen different reports. All reports are not 
created for each evaluation; for instance, the real-time reports would not be 
produced when a batch processing system is being simulated. 

The entire SCERT operation is shown as Figure 6. As can be seen there are 
five functional phases. ^ 

o Phase 1 - The Introduction of Processing Requirements 

The first input to SCERT is a series of definitions outlining the workloads and 
computer processing requirements of the system to be simulated. Specifically the 
information entered is concerned with three areas. 

^Unless otherwise footnoted, this and future information relating to SCERT was 
was generally obtained from A Technical Description of SCERT (Rockville, Maryland: 
Comress, Inc.). 
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SOURCE: A Technical Description of S CERT (Rockville, Maryland: Comress, 
Inc . ) , p . 16. 



Figure 6 — The SCERT Program 
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Systems Environment information relates to the environment in which the 
computer and data processing system will operate. The detail data elements are: 

Cost Data - Specifies the costs of operation such as programmer and 
operator salaries, and systems life expectancy. 

Programmer Experience Profile - the user indicates the experience and 
qualifications of the programming personnel. 

Definition Percentiles - the user indicates the percentage of die total 
system requirements not defined for analysis by SCERT. The simulator uses these 
to project the total. 

Real Time - the parameters concerning real time processing, if any, arc- 

entered. 

File Definition defines each file in the data system. Each file is given a unique 
number; the size of the file is specified by indicating die number of records; die 
number of characters and fields per record and the type of file is indicated. 

Systems Definition is die common denominator for defining the processing 
requirements in the individual computer run or random event. Each run definition 
consists of three distinct levels: 

Run identification, frequency, priority, and prerequisites. Each run is 
defined by a unique number and die frequency is indicated as daily, weekly, monthly, 
quarterly, or yearly. 

File ident ification and through- put parameters. Each file is defined by the 
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unique number used in the file definition above. Any special considerations 
concerning the way the data is to be handled is indicated. 

The internal activity section provides the simulator with the internal 
processing applicable to the input and output files. 

Phase 1 has three primary functions: 

. Accept the input definitions indicated above; 

. Build a mathematical model of each computer run; and 
. Validation of the model and output diagnostic data if any errors or 
inconsistencies are found. 

o Phase 2 - The Introduction of Hardware/Software To Be Simulated 

In Phase 2 the specific hardware and software configuration being appraised is 
introduced. This is a very specific and detailed definition of the hardware. The 
model numbers of every component, all special features, adaptors, number of 
channels, etc. to be used have to be provided. In addition, the details applicable to 
the software package have to be indicated. 

The functions of Phase 2 are: 

. To accept the hardware/ software definitions; 

. To build a mathematical model representing hardware/software; and 
. To validate the compatibility of the models built in Phase 1 with 
Phase 2 models. If discrepancies are found, a diagnostic report is 



created. 
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o Phase 3 - Presimulation Algorithms 

In this phase the models built in Phase 1 are passed against the models of 
Phase 2. During this processing a series of calculations are performed which in 
effect structure and parameter the non-hardware oriented models to the performance 
abilities and capabilities of the hardware. 

During Phase 3 the following events for each program occur. 

. The internal processing time and memory requirements are calculated. 

. All files are assigned to I/O devices and channels. 

. All files are structured to meet the hardware parameters. 

. The thru-put time for all I/O and memory functions is determined. 

. The timing of the software packages is determined. 

. The pre- and post-run times for factors such as setup-time, error- 
correction, etc. are calculated. 

o Phase 4 - Simulation 

In this phase the actual simulation of the system takes place. All of the work 
performed in the first three phases was in preparation of the data for input to Phase 
4. There are three distinct stages: 

Stage 1 - Each program run or random event is expanded into its maximum 
number of unique thru-put iterations. The processing simulates the flow of each of 
these thru-put iterations through the computer configurations. The results are the 



net thru-put timing. 



57 



Stage 2 - This stage is entered only if the defined processing includes 
random- occu ring events. This stage constructs models based on probability theory 
and distributes the occurrences of events in a probabilistic manner. These models 
are then simulated and potential queueing points are determined, and processing 
delays caused by the queues are derived. 

Stage 3 - This stage is entered whenever the hardware/ software complex 
is capable of multi-programming or multi-processing. In the first two stages the net 
elapsed time and capacity requirements were developed. It is the function of Stage 3 
to schedule the concurrent processing of these events. When this is completed, the 
results stored in the mathematical models is ready to be printed out. 
o Phase 5 - Production of Output Reports 

SCERT can provide the user with fifteen standard reports. If requested the 

simulator can be modified to provide nonstandard or special purpose reports. The 

reports that are of particular interest in validation are given below. A complete list 

of the standard SCERT reports is shown in Appendix H. 

Computer Complement Report — Portrays the exact configuration that was simulated 
and provides certain basic cost data about the configuration. 

Central Processor Utilization — Summarizes for each scheduled run the projected 
running time and horizontal memory utilization. 

Cost Summary — Primary cost output which relates the projected utilization of the 
computer to lease, purchase, and maintenance cost considerations. 

Detail Analysis — Produced for each scheduled run or real-time event simulated 
and serves as backup for all other reports. It precisely portrays the utilization, 
timing, and memory requirements derived by the simulation and presimulation 
algorithms for all components making up the run. 
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Hand Timing 

The essence of hand timing is to analyze each and every function that the 
computer system will be required to perform during the running of a program. The 
hardware engineering time for each operation is determined and all of these times 
are added together to derive the program running time. 

This method of timing requires the user to have a detail knowledge and 
understanding' of the programs that are going to be timed, hi addition, a 
standardized method of gathering the required data should be used. This is necessary 
in order to obtain consistency between vendors and limit the number of approximations 
in the timing data. Some suggested timing tables are proposed by Joslin. There 
are several data elements that should be in a detail time table: 

. Program Number and Frequency 
. Input/Output Requirements 
. Central Processor Memory Requirements 
. Non-productive Time 
. Productive Time for Each Component 

In addition to a detail sheet the user should have a summary sheet. On this form the 
user can enter the data from the detail sheets. From this data the user can then 
determine the monthly run time. 



^ “Josl in, 



Computer Selection, p. 83. 
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Alternative Evaluation Methods 

The purpose of the evaluation process is to define which of the several proposals 
submitted is best suited to the organizations needs. Determining which is best 
forces the user to establish some criteria that can be used as a yardstick to 
measure the proposals. The objective of evaluation is usually to select a 
proposed ADP system, capable of accomplishing the future workload at the lowest 
total cost. 

There are several different ways for evaluating the proposals. The alternatives 
range from a policy of not rocking the boat and staying with the current suppl ier to the 
very competitive situation that uses objective criterion in the selection process, hi 
order to serve the purposes of this paper only the systems considered to be objective 
will be reviewed. 

The literature search for this paper revealed three different objective methods 
for the evaluation of vendor proposals: Weighted Scoring, Cost-Effectiveness, and the 
Cost-Value techniques. Cost-Value was reviewed in Chapter III. 



Weighted Scoring 



There are several different weighted scoring systems that can be used. 



13 



1 3 

For the interested reader: Eugene S. Schwartz, "Computer Evaluation and 
Selection," Journal of Data M a nagemen t, VI, No. 6 (1968), pp. 58-62; and, J.A. 
Campise, "A Quantitative Approach to Equipment Comparison," Journal of Data 
Managem ent, I, No. 5 (1963), pp. 12-20. 



60 



For this paper, the system proposed by Mr. Solomon Rosenthal will be reviewed. ^ 

The first step, when using a weighted scoring system, is the establishment 
of the selection criteria. The basis for developing this plan is the systems 
specification of the information system. The user must define those characteristics 
of an ADP system that will be required to satisfy the needs of the organization. The 
specific items included and the relative weight assignment to each item varies based 
on the requirements of the information system. In Mr. Rosenthal’s system, the 
factors are classified as major, intermediate, or minor. 

The major categories include factors such as: overall, costs, equipment 
characteristics, systems potential, vendors support and others as required. The 
intermediate and minor categories would include the component elements that make 
up the major area. For example, within the cost element, details would focus on 
the cost of individual components and features of each, hi addition, costs relating 
to the installation of die equipment and its subsequent operation would also be 
considered in the intermediate or minor categories. 

The next step is to assign weights to the factors to be used to evaluate the 
proposals. It is important to note that if this system is to be impartial, the weights 
must be assigned before any vendor proposals are received. Li fact, the weights 
should be assigned before the RFP is circulated. The maximum score possible is 

^Solomon Rosenthal, "Analytical Technique for Automatic Data Processing 
Equipment Acquisition," Proceedings of the AF1PS Spring Joint Computer Conference , 
1964, pp. 359-366. 
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established as 100. This score can be achieved only if one proposal is the best in all 
factors. A part of the 100 maximum is assigned to each major factor based on the 
relative importance of each. Then weights are assigned to the intermediate and 
minor factors. The sum of the weights assigned to the intermediate and minor 
elements must be equal to the appropriate major factor of which they are a part. 

The results of this process might look as presented in the following example. 



FACTORS 




WEIGHTS 


Cost s - 




35 


Machine Rental 


20 




One-Time Costs 


10 




Personnel Support Cost 


5 




Equipment Capability 




25 


Random Access Storage 


10 




Systems Validity Checks 


5 




Processing Speed 


10 




System Potential 




15 


Ability to Increase 


10 




Random Storage Rapidly 
Ability to Augment 
Central Processor 


5 




Manufacturer Support 




10 


24-Hour On-Site Maintenance 


5 




Back-up in Vicinity 


5 




Common Language Programming 




10 


COBOL 


5 




FORTRAN IV 


5 




Physical Requirements 




5 



After the proposals are received and validated, they are ready for scoring. It 
should be noted that when using this system the individuals involved in the validation 
of the proposals do not do any scoring. The scoring is accomplished by a completely 
different group of people. In addition, the scorers do not know which vendors' 



62 



proposal they are processing. In fact, ihe scorer does not even see the proposals. 
The scores are recorded on an evaluation sheet that was prepared in the validation 
process. The evaluation sheet shows the data applicable to each factor. In addition, 
the validators prepare a written report indicating ihe costs, processing times, and 
differences between the proposals and the specifications. ^ 

Based on this information, the scoring begins with the minor elements first, 
then intermediate scores are developed, followed by the majors and finally the grand 
total. The minor scoresare developed by either of two methods. Some minor factors 
are not compared one proposal with another, but are assigned a predetermined score 
if certain conditions exist. For example, if free maintenance is provided twenty-four 
hours per day, "X" points might be awarded, if only for eight hours per day, a lessor 
score would be assigned. 

The second method would apply to the majority of the minor elements. If the 
smallest is best, the score is computed as follows: 



(comparison base) (maximum possible) 
(this vendor entry) 



minor score 



If the largest is best, the following formula is used: 



(this vendor entry) (maximum possible) 
(comparison base) 



minor score 



> 16 



15 



Ibid. 



16 



Ibid. 
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The next step is to total the minor scores within each intermediate. The 
intermediate scores are added together to form the major scores and the final score 
is the sum of all the major scores. 

The proposal then with the highest score would be recommended to top 
management as being the best system to select. 



Cost Effectiveness 



The concept of cost effectiveness is primarily attributable to the work of Hitch and 

McKean, and the principlesof this concept provide the foundation for a computer 

selection system developed by the Mitre Corporation. Prior to analyzing this system 

1 R 

however, it will be helpful to define the words cost and effectiveness. 

Effect iveness - The degree to which a system will perform the future jobs and 
satisfy the constraints. Effectiveness is generally considered to consist of the 
following three main components. . . . 

a. Capability - The degree to which a system will perform the future jobs and 
satisfy the constraints, assuming that the system is always available for operation 
and will never malfunction. Capability can be measured in various ways but the 
two key measures of capability are: 

(1) Quality of the work output. This measure is in general multi- 
dimensional since it encompasses the many sub measures used to measure 
the work output. For example, it might include the straightness of a line of 
print or the maximum number of copies of printout. 



Charles J. Hitch and Roland N. McKean, The Economics of Defense in the 
Nuclear Age (Cambridge, Mass. : Harvard University Press, 1963). 

^J.D. Porter and B. H. Rudwick, Appli cation of Cost - Effectiveness Ana l ysis 
to EDP System Selection (Springfield, Virginia Clearinghouse: U. S. Department of 
Commerce, AD 667-522, October, 1967). 
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( 2 ) Time . Given that the quality of the work output can he measured, a 
second key measure of the EDP system capability is the time taken to 
perform the future work load, again assuming the system is available for 
and never malfunctions. For example, the measure could be the expected 
time to perform a given monthly workload. 

b. Availability - may be defined as the probability that the system will be ready 
for operation when called upon. 

c. Defindability - may be defined as the probability of the system completing 
the job satisfactorily, given that it was available. . . . 



Cost - The total dollars required to procure, operate, and maintain the system 
to perform the future set of E DP jobs. . . .19 

Since these two elements have been defined it is possible to measure each one 
separately. The problem then becomes one of combining these two elements in a 
way to facilitate decision making. To do this it is necessary to specify in advance the 
level of either the cost or the level of the effectiveness. The mechanics are explained 
below and illustrated in Figure 7. 

Level of effectiveness specified: called pivoting on constant effectiveness. If 
level E 2 is selected as the comparison level, system A could not be selected. If the 
effectiveness of A were increased to E 2 its cost would then be at either point A 3 or 
Ag. Now system A and B can be compared with each other. If the A system cost 
were A^ then it would be selected; however, if the cost were at A 3 then the 13 system 
would be selected. 

Level of cost specified : called pivoting on constant cost. If the C 2 cost level is 
selected as the comparison level of cost, and the cost of system A is increased, 
its level of effectiveness moves to either A 2 or A 4 . When A's effectiveness and cost 



19 



Ibid. , p. 13. 
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SOURCE: J.D. Porter and B.H. Rudwick, Application of Cost-Effec tiv eness 
Analysis t o EDP System Selection , U.S. Department of Commerce, AD 667-522. 



Figure 7 — Measurement of Cost and Effectiveness 
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intersect at the B system is the better buy. The converse is true when the 
effectiveness and cost of the A system is represented by the point A2. In this 
situation the A system would be the better buy. 

The first selection criterion, pivoting on constant effectiveness, is more 
applicable to the problem of computer selection, so the cost-effectiveness system 
devised by Porter and Rudwick is based on this principle. u 

There is a great deal of similarity between the cost-effectiveness system and 
the cost-value system used by the Navy. Both methods are based on: a probabilistic 
workload projection, mandatory requirements, and desirable features. 

There are differences, however in the way the projected workload is derived and 
in the way that value is calculated for the desirable features. Both systems however 
regard the mandatory items in the same way. The mandatory features are validated 
and die requirements in the RFP concerning these items must be satisfied for the 
proposal to be considered responsive. 

The total selection process using the cost- effectiveness system is shown in 
Appendix I. 

Probabilistic workload projection: the basis for using probabilities in 

projecting future workload is the fact that the user can predict with a high degree 

of confidence certain "known jobs”; however, there arc "likely jobs" which can 

2 1 

only be expressed quantitatively with a lower degree of confidence. 



20 
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Ibid. , p. 17. 



Ibid. , p. 18. 
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The user prepares a projection of his future workload by starting out with a 

diagram similar to the one in Figure 8. The lowest line is called the "Reference 

Workload" and the user lias indicated that there is a 100 per cent probability that this 

workload level will be experienced or exceeded. The user then assigns a probability 

to each of the other workload level projection lines. 

The next step is to average the yearly workload levels. This, then, permits a 

diagram like that shown in Figure 9 to be drawn. The purpose of this is to simplify 

22 

the cost calculations. 

In the validation phase, timing data was developed for each system proposed. 

This information is correlated with the various workload levels. If benchmarks were 
used this correlation is accomplished by using the extension factors. 

The evaluation team can now construct a chart similar to the one shown in 
Figure 10. In this situation the RFP had a mandatory requirement that the workload 
be completed in 600 hours. In view of this, Vendor Abas proposed two systems: A] 
and A 2 . System A^ , a cheaper system, can satisfy the mandatory requirement until 
sometime in the third year of the system life. Then it is augmented to become system 
h- 2 , which can satisfy the 600 hour requirement for the remainder of the system life. 

It will be noticed in Figure 10 that the method of expressing the probability is 
different. The figures now represent the probability of the workload falling within the 
respective segments. 



22 



Porter and Rudwick, Cost Effectiveness, p. 23. 
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P 5 = .0 




SOURCE: J.D. Porter and B. 11 . Rudwick, App lication of Cost- Effe ctive ness 
Analysis to EDP S ystem Selec tion , U.S. Department of Commerce, AD 667-522. 



Figure 8 — Probabilistic Workload Projection Smoothed Over Time 
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Maximum Workload 




Operational Year 



SOURCE: J.D. Porter and B. H. Rudwick, Applicati on of C ost-Eff ectiveness 
Analysis to EDP System Selection, U.S. Department of Commerce, AD 667-522. 



Figure 9 — Probabilistic Workload Projection Averaged Per Year 
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HOURS OF USE 




SOURCE: J.D. Porter ynd B. H. Ruthvick, Application of Cost-Effe ctiveness 
Analysis to EDP System Selection, U.S. Department of Commerce, AD 667-522. 



Figure 10 — Projected Time for System Aj and A. } Proposed By Vendor A 
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The cost of each system can now be calculated. This is done by determining 
the yearly cost of operating each system as follows: 

C = Total Cost 

P^ = Probability of workload in 

segment 1 

C ^ = Cost of operation in segment 1 

C = PjCj+P^T. . . ,+P n c n 

After the yearly cost is determined the total discounted present value is derived in a 

9 Q 

manner similar to the one shown in Appendix J. 

Dcsii'able features make up the third category of the user's data processing 
requirement. Porter and Rudwick are concerned with the necessity of this third way 
of expressing data process requirements because: 

1. The representative workload only approximates the actual workload. 
Since there may be other workload requirements that will not be measured in the 
systems timing determination, by including desirable features the user is provided 
a hedge against this uncertainty. 

2. The inclusion also provide a hedge against the uncertainty attendant to 
measuring a systems ability to perform the future workload. ^ 



23[J.S. , Executive Office of the President, Bureau of the Budget, Discount Rates 
and Procedure s to be Used in Evaluating Deferr ed Costs a nd Benefits, Circular A- 94, 
Washington, D.C., Attachment A. 

"‘^Porter and Rudwick, C ost Effectiveness , p. 18. 
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3. The use of desirable features permits a reduction in the number of 
mandatory requirements. By definition the vendor must satisfy all mandatory items 
or his proposal is eliminated. This could create an undesirable situation. For 
example, suppose the RFP contained a mandatory requirement for the proposed 
system to have a two second response time. Vendor A's system has a response 
time of 2. 1 seconds, but that system costs 20 per cent less than all others. It is 
difficult to defend the need for that additional one-tenth of a second advantage, 
particularly in view of system A's cost. 

4. The manufacturers of data processing equipment are competing with each 
other. This spawns technological innovations which improve the design of computers 
and the attendant software. The use of desirable features in the RFP provides the 
vendor an excellent avenue for proposing these advances to the user. Without these 
features the user would be denied the opportunity of evaluating these advanced 
capabilities. 

Once the desirable features have been stated it is then necessary to provide a 
way to evaluate the vendor proposals. There has to be a way to relate the character- 
istics of any particular desirable feature to the job or jobs that will be improved by 
the use of this desirable. The question then simply stated is: Is the desirable feature 
offered by the vendor worth the price he charges? This question cannot be answered 
unless a value of this worth has been assigned. In order to assign a value, however, 
the term worth has to be defined. In this system, the worth of a feature is equal to 
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"die lowest incremental cost to do the same job if the feature is not available. "25 

There are two ways provided for determining and evaluating the worth of any 
desirable feature. The first way determines worth through an analytical process 
and the second method makes use of comparative ranking. 

Analytical determination of worth: Since the worth of a feature is defined as 
the lowest incremental cost of doing the job if the desired feature is not available 
from the suppliers, it is necessary to first identify the alternative ways of doing the 
job. Figure 11 depicts the alternatives that are available to the user. The next 
step is to determine the cost over the life of the system of alternative ways of 
accomplishing the function. The worth of the' feat ure is the least of these costs. 

It is also necessary to identify the cost of the alternative ways of obtaining the 
feature and doing the job using the feature. For example the feature might be procur 
procured from other sources or, in the case of software the user might develop the 
capability using in-house resources. 

Based on this compilation of data, it is then possible to determine the way to 

accomplish the function at the lowest total cost. This is referred to as the "efficient" 
26 

solution. Restated, the efficient solution is found by determining the lowest cost 
method of obtaining the feature and doing the job, then comparing this cost with the 
worth, and choosing the lowest cost alternative. 



25 



26 



Ibid. , p. 33. 



Ibid. , p. 34. 
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Do Job 




Not Available Available 




V 




Least Cost = Worth 



V 




Least Cost of Feature 



SOURCE: J.D. Porter and B.H. Rudwick, Application of Co st - E ffe c t i ven e s s 
Ana l ysis to ED P System Selection^ U.S. Department of Commerce, AD 667-522. 



Figure 11 — Alternatives Available to the User 
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Determination of worth by comparative ranking: There may be times when it 
will not be possible to determine the worth of all desirable features by analysis and 
considered judgement. In these instances, intuitive judgement is substituted as a 
part of the quantitative evaluation. The steps to implement this procedure are: 

(a) All desirable features are ranked in order of importance. 

(b) Points are allocated to each feature thus establishing the relative worth of 
each. 

(c) The points are translated into dollars. This is done by calibrating one or 
more of the features by determining its worth on the analytical basis described 
above . 

(d) The results derived are reviewed using intuitive judgement as the sounding- 
board. If discrepancies are found then either all or some of the items should go 
through the whole process again. 27 

It is now time to bring together the information derived in the various areas, 
so that a composite picture of each proposal -can be developed. To do this a 
worksheet like the one shown in Figure 12 will be used. In this example there 
are three vendors' (A, B, and C) proposals being evaluated. There are three 
desirable features to be considered — Fi, F2, and F3. The users worth and the 
least cost for each is shown in item 3 on the Evaluator Worksheet (Figure 3 2). 

This illustration demonstrates how the costs are derived if the vendor does not 
provide the feature at all, or the feature is provided in the basic system and cannot 
be excluded, and finally the instance where the vendor does propose the feature and 
states its cost. 

The initial entry on the worksheet is the Total Proposed Vendor Cost. This 



27 lbid., p. 37. 
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Cost Elements 


System Cost 


















c A 


c B 


c c 


i. 


Total Proposed Vendor Cost 






300K 


31 OK 


3301C 


2. 


1 


ixpected Cost to Do Representative Workload 


300K 


305K 


31 OK 


3. 


Cost of Additional Job Benefits 










! 




i * 

Desirable User 


Least 


Vendor Cost 












Feature 


Worth 


Cost 


_Ca_ 


'Cr 


c c 












F 1 


10K 


15K 


- 


incl 


15K 


10K 


- 


10K 






f 2 


25K 


20K 


- 


- 


5K 


2 OK 


2 OK 


5K 






F 3 


10K 


2QK 




5K 


incl 


JOK 


1 

5K i 




4. 


Total Expected Cost To Do User Job 




34 OK 


J 

330K 

I 

i 


325K a 



^Vendor C selected - lowest total cost. 



SOURCE: J.D. Porter and B. H. Rudwick, Application of Cos t-Effectiveness 
An alysis to EPP System S election, U.S. Department of Commerce, AD667-522. 



Figure 12 — Evaluator's Worksheet 
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represents the cost of each manufacturers proposed system, if the system is 
accepted as proposed. It also is the cost as calculated by the vendor. 

The evaluator enters the previously derived cost data to process the workload 
of the life of the system, and thus derives the expected cost to do a representative 
workload. 

At this time the desirable features which each vendor has proposed and die 
incremental costs associated with each are entered on the worksheet to determine 
the cost of additional job benefits. In this example note that vendor A does not 
provide any of the three features. In the case of vendor B, feature is included in 
the system cost while F2 is not provided but F3 is at the cost indicated. Vendor C 
has provided all of the features. The cost of feature F3 however, is not quoted 
separately but included in the system cost. Now the evaluator can determine for 
each vendor the least costly of the three alternative ways of receiving the benefits 
provided by each of the desirable features. The three alternatives are: 

a. Buying the desirable feature from the vendor (at the vendor's proposed cost). 

b. Obtaining the desirable feature from another source (at the least cost of 
feature if obtained separately). 

c. Not buying the feature, but using the least costly alternative way to provide 
the benefits (at a cost equal to user worth). 28 

The evaluator now enters the values representing Users Worth and the Least 
Cost. These values arc calculated prior to the actual evaluation process. 

The lowest additional user cost for obtaining the desirable feature can be 



^Porter and Rudwick, Cost Effectiveness, p. 45 . 
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determined. Note in Figure 12 the cost data entered for the three desirable features. 
In the case of Fj the User Worth is $10K, that is the user can perform the jobs 
associated with this feature at an expected cost of $10K. Since vendor A does not. 
provide this feature, the user will be forced to spend $30K in addition to vendor A 
costs to satisfy the requirements associated with Fj. Vendor 13 includes this feature 
as part of his basic system and the cost of it cannot be segregated. The user then 
will not have to spend the $10K if vendor B's system is used. Vendor C can provide 
Fj however, the cost is y $15K. Thus, the evaluator would eliminate the feature from 
vendor C's proposal since its cost is higher than its worth to the user. This same 
type of analysis is applied to all of the other desirable features. 

The total expected cost to the user is then calculated by adding the cost of 
the desirable feature, or the user cost equivalent, to the expected cost to do the 
representative workload. This cost completes the cost calculation. 

There are some points worth noting concerning this method of arriving at the 
final cost figures for the proposals. Vendor A had the lowest proposed cost and the 
lowest cost to perform the representative workload. On the other hand, winning- 
vendor C had the highest proposed cost and the highest cost to perform the workload. 
These costs however, do not cover all of the users requirements. The total cost is 
the only one that will provide for satisfying the total data processing requirement of 
the user. Accordingly the total cost figures are the true criterion for making the 
decision . ^ 

29 



Ibid. , p. 46. 



CHAPTER V 



EVALUATION OF THE NAVY SYSTEM 



II is the purpose of this chapter to evaluate the Navy's compelive computer 
selection system. The review will be conducted by first appraising the organization 
responsible for selection and then by looking at the various stages in the process. 

The Automatic Data Processing E quipment Select ion Office 

The Secretary of the Navy established ADPESO on 1 July 1967. It is essential 
to appreciate the role played by this organization if one is going to evaluate the 
Navy's computer selection system. This becomes abundantly clear when one 
considers the fact that prior to the establishment of ADPESO there was no one 
system in use. The selection of ADPE prior to ADPESO was accomplished by 
the various heads of departmental components. The procedures used varied 
depending on the philosophy of each of the departmental components. There was 
widespread use of ad hoc committees that were formed as the need arose. The 
contracting wa s conducted by a regular contracting officer, not one specialized in 
the procurement of ADPE. 
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The establishment of ADPESO thus accomplished two very significant things. 
First the responsibility for selection of ADPE was centralized in the Navy. Secondly 
a full time staff was recruited to carry out the mission of ADPESO. 

In addition to centralizing the function, the function was also elevated to a 
higher level in the Department of the Navy. The Director of ADPESO reports to the 
Special Assistant to the Secretary of the Navy. Thus, top management attention is 
always focused on this vital function. 

There arc twenty- two professional members on the staff of ADPESO. These 
individuals are responsible for developing the procedures used, plus performing the 
functions of equipment selection. There are several significant benefits derived from 
this. First, and probably most significant, is the fact that these individuals are 
professionals. This means that the breadth and depth of knowledge of each individual 
presently involved in the selection process is many times greater now than was the 
case prior to July 1967. This increased knowledge has a very profound impact on the 
RFP's sent to vendors and the appraisal of the proposals submitted by the suppliers. 

In the years prior to the establishment of ADPESO, the specifications and RFP's 
were prepared by the user activities. The degree to which these specifications were 
not oriented toward any given manufacturer was dependent on the experience of the 
individuals preparing them. Since these individuals were normally not involved fuli 
time in the selection process, their range of knowledge was limited. In view of this 
it was possible for the specifications to be oriented one way or another. The full time 
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professional .staff at ADPESO does not labor under this handicap so the package sent 
lo the vendors is not susceptible to this flaw. 

When the vendor’s proposals are received, the full-time professional staff is 
responsible for reviewing them. From time to time, user personnel do assist in the 
appraisal process, but this effort is still under the close control and direction of 
ADPESO. As indicated in Chapter III, the group actually responsible for preparation 
of the RFP and the review of all the corresponding proposals is the Source Selection 
Evaluation Board (SSEB),. It is important to note that the SSEB is constituted for 
the entire life of equipment acquisition. This means the individuals can become 
intimately acquainted with the requirements of the individual cases. This 
comprehensive understanding of the user requirement coupled with specialized 
knowledge of the individuals on the SSEB create a climate favorable to rational and 
objective evaluations. 



Case 


GSA 

Price 


ADPESO 

Negotiated 

Price 


Dollar 

Savings 


% 

Discount 


A 


$3,300,000 


$2,600,000 


$ 700,000 


21% 


B 


4,743,612 


4,327,981 


415,631 


9% 


C 


5,861,976 


3,752,796 


2, 190, ISO 


37% 



SOURCE: Information provided by Mr. A.E. Feenan, Head, 
Specification and Proposal Division, Automatic Data 
Processing Equipment Selection Office. 



Figure 12 — Navy Savings by ADPESO 
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The contracting officer is a key ADPESO employee. He devotes all his efforts 
to contracting for ADPE. By specializing in the procurement of ADPE, this individual 
has developed an in- depth knowledge, understanding, and appreciation for the 
particular problems associated with acquisition of the equipment. The results are 
substantial savings for the Navy as shown in the cases presented in Figure 12. 

Evaluation of Selection Procedures 

In Chapter III the Navy's selection process was divided into five subcategorics. 
Accordingly, the evaluation of the system will be done by examining each of these 
areas. 



Systems Specifications 

First, it is significant to point out that prior to the establishment of ADPESO, 
there was no standard method by which a user could document his data processing 
requirement. Each level in the user's chain- of- command had its own unique set of 
requirements concerning what information should be included and the format in which 
the specifications were to be presented. Accordingly ADPESO' s most important 
initial contribution to computer selection in the Navy was to promulgate instructions 
that standardized the format of the specifications. 1 

^E. O. Joslin, "Competitive Computer Selection Within the Department of the 
Navy," (Unpublished paper presented to the Dicbold Group, Inc. , October, 1969), 

p. 5. 
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The specification review concluded by ADPESO is very extensive. This review 
plus the standardized format provides the Navy with an objective set of requirements 
for submission to the equipment suppliers. 

It is significant to note that it is the Navy's policy to minimize the number of 

O 

mandatory requirements contained in the specifications. Every item that is 
classified as mandatory is carefully reviewed. When the justification does not 
substantiate the claim the item is reclassified as a desirable feature. This is 
done so that the specifications will not become so stringent as to either discourage 
vendor participation or to cause an undue number of proposals to be classified as 
non- responsive . 



The Selection Plan 



The selection plan consists of the detailed instructions that will be used to 
evaluate the proposals submitted by the vendors. The significant feature of the 
Navy's philosophy here is that the plan must be prepared before the RFP is issued. 3 
This means that greater objectivity is injected into the evaluation process. If the 
plan were not prepared until after the initial review of the proposals, there would 
be danger of personal preference creeping into the plan. 



2(J. S. , Department of the Navy, "Specifications, Selection and Acquisition of 
Automatic Data Processing Equipment, Secretary of the Navy Instruction 10162. 13, 
Department of the Navy, Washington, D.C. 20390, p. B-ll. 



3 Ibid. , p. Ill- 5. 
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Vendor Liaison 



The ADPESO has defined some very meaningful rules that govern the 
relationship between its representatives and the various manufacturers. The intent 
of these rules is to establish an impartial and objective way to deal with the vendors. 
These niles have been provided to the vendors so that they know how to proceed in 
their dealings with the Navy personnel. One very noteworthy policy is that 
information provided to one supplier is provided to all and in writing. The 
comprehensiveness of the policies concerning supplier relations plus the fact 
that ADPESO is a continuing body staffed with professionals creates a climate for 
fair and objective interface between the vendors and the Navy. 



Validation of Vendor Proposals 

The validation of the vendor’s proposals is concerned with two areas. First is 
the verification of the proposed systems capabilities and die second is validation of 
the system timing. The Navy uses several sources of data to verify die systems 
capabilities. First, the vendors are requested to provide technical data in their 
proposal. In addition ADPESO has manufacturer teclinical manuals and die Auerbach 
Standard EDP Reports. There is however, a factor that seems to be even more 
important than these aids- -the professional staff at ADPESO. Because these people 
are continually dealing with the subject matter, dieir ability to appraise the vendor 
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claims is certainly greater than what could have been expected prior to the 
establishment of ADPESO. 

The primary method of validating the system timing is through benchmark 
programs. In Chapter IV two other methods of system timing were discussed- 
computer simulation and hand timing. There are advantages and disadvantages to 
each of these methods. In certain instances it is possible that any one would be the 
best considering all factors. Joslin believes however, that the prime consideration in 
choosing a timing technique should be the degree of confidence required in the data 
derived. The following chart indicates die levels of confidence that can be obtained 
with the three methods. Notice that the confidence levels are shown covering a wide 
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range. This is necessary lo allow for variations in the applications and personnel 
using the techniques. The information indicates that the highest levels of confidence 
can be placed in the results obtained from the use of a benchmark program.^ 

There is another significant advantage of using benchmark programs — the 
programs arc run on the system proposed by the vendor. This gives the user an 
opportunity to determine the effect on the application of such features as: I/o control, 
central processor administrative and processing functions, effective speed of the 
various I/o devices, system simultaneity, the software scheduler, and the efficiency 
of the programming system in using the hardware capabilities. ^ 

Another very important item to consider is the cost element. The U.S. Army 
in 1968, conducted a study designed to determine the advisability of continuing an 
effort to program a computer simulator. In the course of this review, cost data was 
developed concerning three timing techniques. The Army found the cost per project 
for obtaining timing data, was: 

Hand Timing $34, 124, 

Simulation $78,532, and 

Benchmarks $29, 548. ^ 

4 

E.O. Joslin, Computer Selection (Reading, Massachusetts: Addison-Weslcy 
Publishing Company, 1968), p. 105. 

5 Ibid . , p. 78. 

^U. S. , Department of the Army, Computer System Support and Evaluation 
Command, Report of the S'^ Study Group System and Software Simulator (Washington, 
D. C. : December 20~~ 1968). “ 
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Based on this plus oilier considerations in the study, the Army decided not to 
continue developing a simulator. Instead, benchmark programs are being used. 

There are disadvantages to using the benchmark programs however. 

Before benchmark programs can be used the actual programs have to be written. 

This task takes time. The amount of time varies with the complexity of the 
programs, the knowledge and experience of the programmers, particularly the level 
of experience with programming languages such as COBOL, FORTRAN, or JOVIAL. 
This is so because these languages generally can be processed by all the different 
computers. This time clement could cause the user a loss of a state-of-art condition. 
The influence of this factor should decrease within the Navy in the future for the 
trend is toward using these higher level languages as a matter of course. In light 
of the advantages stated pcrviously, plus the fact that the DOD now requires all 
services to use benchmarks, it appears that the Navy's use of this technique makes 
the best of a difficul t task. 



Evaluation of Vendor Proposals 

The method used by the Navy to evaluate vendor proposals is based on the Cost- 
Value Technique. In Chapter IV, two other evaluation systems were reviewed — 
Cost-Effectiveness and the Weighted Scoring methods. There is a great deal of 



7 

Joslin, Computer Selection, p. 18. 
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similarity between the Cost-Value and Cost-Effectivencss systems. One difference 
is the way the worth of a desirable feature is derived. Another difference, only slight 
however, is in the way the workload projection is handled. 

The Cost-Effectiveness system does not pi'ovide for the probability of the 
workload falling below the projected reference workload line. In the Navy's system, 
the user indicates a projected rate of growth and then assigns a probability to the 
levels above and below the trend line. This approach seems to be more realistic. 

The significant advantage of the Cost-Value system versus the Weighted 
Scoring method, particularly to lop management, is that the results are expressed in 
dollars instead of points. By expressing the evaluation of each vendor proposal hi 
dollars, the senior managers can readily understand what is happening in the 
evaluation pi'oeess, but; there is no real need for these people to understand complex 
and technical data concerning each proposal. 

Another important characteristic of the Cost-Value technique not found in tire 
Weighted Scoring system is the ability of the user to analyze cost associated with 
desirable features. The user can look at worth and cost of a desirable feature and 
make a sound procurement decision. This capability is particularly important in light 
of the new pricing policies of ADPE manufacturers. 

While the Weighted Scoring system appears to be objective, there is a very 
significant crack in its wall of objectivity. The factors chosen as the evaluation 
criteria and the assignment of weights to them is subjective not objective. There are 
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no objective guidelines to assist buyers in matching requirements with factors or the 
relative importance of each. It is unlikely that any two analysts, given the job of 
independently establishing appropriate factors and weights to select the best computer 
for a particular installation, would arrive at similar conclusions. There also is a 
possibility that the weights could be juggled and thus lead to almost any result. 
Further, there is a real danger that by the time the analyst has performed all the 
necessary weighting and scoring, he might lose sight of their shaky subjective 

O 

foundation and attach undue significance to the results. 

In view of the above factors, it is believed that the Cost -Value technique is the 
best method of evaluating the vendor's proposals. 

Suggested Improvement 

While the Navy’s selection system is excellent, there is one area that could be 
made even more meaningful. Currently the method of costing the vendor proposal 
results in a figure that is the most probable. This figure is derived by the method 
described in Chapter III. Basically the calculations involve a monthly workload cost 
at various levels plus the probability of operating at these levels and a discount factor. 

Mr. David Hertz proposes a way of informing the executive of the uncertainity 
surrounding capital investment decisions. 9 In this system, probabilities are assigned 

8j 0 hn R. Hillegass, "Systematic Technique For Computer Evaluation and 
Selection," Management Services , VI (Ju! y- August, 1969), p. 39. 

9 David B. Hertz, "Risk Analysis in Capital Investment," Harvard Busine ss 
Review, XXXXII No. 1 (January/February 1961), p. 95. 
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the various possible outcomes. The cash flows connected with each of these 
possibilities is calculated. These values are then used to determine die expected 
net present value. This data is then portrayed graphically. The significant 
advantage of this is that it allows management to sec the expected cash flows plus 
the variability of the flows and the risk associated with each. The present system 
contains all of the data required to perform such an analysis. It would be necessary 
however, to obtain a computer program to perform the required calculations. If this 
type of analysis were incorporated into die Navy system it is believed that dierc would 
be substantial benefits derived. Accordingly it is recommended the Navy conduct a 
study and investigate die feasibility of using a-risk analysis technique in its evaluation 
process. 

Since ADPESO was established, experience with centralized competitive 
selection of computer systems in the Navy has demonstrated that: the using activities 
are happier, the vendors arc happier because they feel they are being given a true 
chance to bid, and the cost of selection within the Department of the Navy has been 
reduced 50 per cent while the number installations has increased 30 per cent. Thus, 
centralized, competitive computer selection in the Navy has proven to be very 
satisfactory. 



^joslin, "Competitive Computer Selection Within the Department of die Nav^ 



p. 35. 



CHAPTER VI 



CONCLUSION 

The purpose of this paper has been to appraise the competitive computer 
selection system used by the U.S. Navy. First, the Navy's system was defined and 
then alternative methodologies were reviewed in order to have a basis for the 
appraisal . 

The Navy's competitive computer selection system is a highly regulated process. 
There are many regulations and policies issued by higher authorities such as the 
Congress, the President, the Secretary of Defense and others. 

The Navy does however, have a great deal of latitude in deciding exactly which 
methodologies will be used to differentiate among vendor proposals. 

The Automatic Data Processing Equipment Selection Office is the organization 
within the Department of the Navy charged with the responsibility for the development 
and administration of the Department of the Navy plans, policies, procedures, and 
methods governing ADPE specification preparation, proposal solicitation, proposal 
evaluation and selection. 

The ADPESO is located in the office of the Special Assistant to the Secretary of 
the Navy, who is designated as the senior ADP policy official in the Navy. 
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The Navy' s competitive computer selection process consists of five major 
activities: 

. Specification Review 

Preparation of the RP’P and Selection Plan 

. Vendor Liaison and Validation of Proposals 

Proposal Evaluation 

. Contract Negotiation and Award 

The timing of the proposed computer configurations is validated by using 
benchmark programs. In conjunction with these programs, a probabilistic workload 
projection technique is used to define the monthly cost to perform die workload. The 
monthly costs are consolidated into yearly increments to arrive at the annual costs 
which in turn are discounted to represent the present value of the total cost to 
perform the workload over the total system life. 

The Cost- Value Technique is used to evaluate the desirable features offered by 
the vendors. The distinguishing feature of this technique is the assignment of value 
to desirable features in terms of dollars. Thus, an important benefit derived from 
the use of this method is improved management understanding of the evaluation 
process, thereby facilitating meaningful decisions relative to the value of any or all 
of the desirable features. 

By contrast, the Navy could validate the timing of the proposed systems by 
calculating the times manually or by using a computer simulation routine such as 
SCERT. These methods however do not provide data that is as reliable as the bench- 
mark programs. In addition, a cost study conducted by the U.S. Army found that 
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benchmark programs could be prepared and used at a lower cost than the other two 
techniques. 

The Cost- Effectiveness and Weighted Scoring systems are alternative processes 
that can be used to evaluate vendor proposals. The Cost-Effectiveness and Cost- 
Value systems are very similar. There is a slight: difference in assigning worth to 
desirable features. There is a marked difference between the Cost- Value and the 
Weighted Scoring techniques. The most significant is that Cost-Value uses dollars 
as an evaluation scheme, .while the latter awards points based on a predefined point 
scoring templet. These scores are added up, and the system with the most points 
ostensibly is the best choice. 

The flaw in Weighted Scoring system is that the factors to be evaluated and the 
weights assigned to them are by necessity, chosen arbitrarily. There are no 
objective guidelines for matching factors and weights to a particular users needs. 

The Author has made one recommendation to improve the Navy system. It has 
been suggested that a methodology advocated by David Hertz, called Risk Analysis 



be incorporated hi the system. 
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To provide for the economic and efficient purchase, leawo, maintenance, opera- 
tion, and utilisation of automatic data processing equipment by Federal 
departments and agencies. 

Be it enacted by the Senate and House oj Representatives of the 
b rated States of Am-erica in Congress assembled , That title 1 of the Automatic) data. 
Federal Property and Administrative Services Act of 1940 (G3 Stat. processing 
377), as amended, is hereby amended by adding a new section io read oquipment. 



“Sec. 111. (a) The Administrator is authorized and directed to 
coordinate and provide for the economic and efficient purchase, lease, 
and maintenance of automatic- data processing equipment by Federal 
agencies. 

“(b) (1 ) Automatic data processing equipment suitable for efficient 
and effective use by Federal agencies shall be provided by the Adminis- 
trator through purchase, lease, transfer of equipment from other 
Federal agencies, or otherwise, and the Administrator is authorized 
and directed to provide by contract or otherwise for the maintenance 
and repair of such equipment. In carrying out Ff is responsibilities 
under this section the Administrator is authorized to transfer auto- 
matic data processing equipment between Federal agencies, to provide 
for joint utilization of such equipment by two or more Federal 
agencies, and to establish and operate equipment pools and data 
processing centers for the use of two or more such agencies when 
necessary for its most efficient and effective utilization. 

“(2) The Administrator may delegate to one or more Federal 
agencies authority to operate automatic data processing equipment 
pools and automatic (lata processing centers, and to lease, purchase, 
or maintain individual automatic data processing systems or specific 
units of equipment, including such equipment used m automatic data 
processing pools and automatic data processing centers, when such 
action is determined by the Administrator to ne necessary for the 
economy and efficiency of operations, or when such action is essential 
to national defense or national security. The Administrator may 
delegate to one or more Federal agencies authority to lease, purchase, 
or maintain automatic data processing equipment to the extent to 
which he determines such act ion to be necessary and desirable to allow 
for the orderly implementation of a program for the utilization of 
such equipment. 

“(c) There is hereby authorized to be established on the books adp fund, ost&b- 
of the Treasury ail automatic de.ta processing fund, which shall be lishment. 
available without fiscal year limitation for expenses, including per- 
sonal services, other costs, and the procurement by lease, purchase, 
transfer, or otherwise of equipment, maintenance, and repair of such 
equipment by contract or otherwise, mxjessary for the efficient coordi- 
nation, operation, utilization of such equipment by and for Federal 
agencies: Provided , That a report of equipment inventory, utiliza- Equipment aoqui- 
tion, and acquisitions, together with an account of receipts, disburse- sit ionu end util 
meats, and transfers to miscellaneous receipts, under this authoriza- iz&tion. 



as follows: 



“automatic OAT a processing equipment 



Punches a and 
utilization. 

5 USC 630.-630g-l. 
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40 USC 481, 474 
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tion shall be made annually in connection with the budget estimates to 
the Director of the Bureau of the Budget and to the Congress, and 
the inclusion in appropriation acts of provisions regulating the opera- 
tion of the automatic data processing fund, or limiting the expendi- 
tures therefrom, is hereby authorized. 

“(d) There arc authorized to be appropriated to said fund such 
sums as may be required which, together with the value, as determined 
by the Administrator, of supplies and equipment from time to time 
transferred to the Administrator', shall constitute the capital of the 
fund: Provided , That said fund shall be credited with (1) advances 
and reimbursements from available appropriations and funds of any 
agency (including the General Services Administration), organiza- 
tion, or contractor utilizing such equipment and services rendered 
them, at rates determined by the Administrator to approximate the 
costs thereof met by the fund (including depreciation of equipment, 
provision for accrued leave, and for amortization of installation costs, 
but excluding, in the determination of rates prior to the fiscal year 
1907, such direct operating expenses as may be directly appropriated 
for, which expenses may be charged to the fund and covered by 
advances or reimbursements from such direct appropriations) and 
(2) refunds or recoveries resulting from operations of the fund, 
including the net proceeds of disposal of excess or surplus personal 
property and receipts from carriers and others for loss of or damage 
to property: Provided further , That following the close of eiHi fiscal 
year any net income, after making provisions for prior year losses, 
if any, shall be transferred to the Treasury of the United. States as 
in isce 1 1 a neons recei pts. 

“(e) The proviso following paragraph (4) in section 201(a) of this 
Act and the provisions of section 002(d) of this Act shall have r.o ap- 
plication in the administration of this section. No other provision 
of this Act or any other Act which is inconsistent with the pro\ isions 
of this section shall he applicable in the administration of this section. 

“(f) The Secretary of Commerce is authorized (i) to provide 
agencies, and the Administrator of General Services in the exercise 
of t lie authority delegated in this section, with scientific and techno- 
logical advisory services relating to automatic data processing and 
related systems, and (2) to make appropriate recommendations to 
the President relating to the establishment of uniform Federal auto- 
matic data processing standards. The Secretary of Commerce is 
authorized to undertake the necessary research in the sciences and 
technologies of automatic data processing computer and related sys- 
tems, as may be, required under provisions of this subsection. 

“(g) The authority conferred upon the Administrator and the 
Secretary of Commerce by this section shall be exercised subject to 
direction by the President and to fiscal and policy control exercised 
by the Bureau of the Budget. Authority so conferred u]x>n the 
Administrator shall not be so construed as to impair or interfere with 
the determination by agencies of their individual automatic data proc- 
essing equipment requirements, including the development of specifi- 
cations for and the selection of the types and configurations of equip- 
ment needed. The Administrator shall not interfere with, or attempt 
to control in any way, the use made of automatic data pnx'essiug 
equipment or components thereof by any agency. The Admin isfm- 
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October 30, 1965 - 3 - Pub. Law 89-306 

79 ST AT. 1129 

tor shall provide adequate notice to all agencies and other usd's con- iio-tioe to 
corned with respect to each proposed determination specifically affect- age notes . 
ing them or the automatic, data processing equipment or components 
used bv them. In the absence of mutual agreement between the 
Administrator and the agency or user concerned, such proposed deter- 
minations shall be subject to review and decision by the Bureau of the 
Budget unless the President otherwise directs/’ 

Approved October 30, 1965. 




Use and Management of Computer 
Technology 

The President's Memorandum to Heads oj Departments 
and Agencies, jinn 28, 1966 

I want t lie head of ev ery Federal agency to explore and 
apply all possible means to 

— use the electronic computer to do a better job 

— manage computer activity at the lowest possible cost. 
I want my administration to give prioiity emphasis to 
both of these objectives— nothing less will suffice. 

The electronic computer is having a greater impact on 
what the Government does and how it docs it than any 
other product of modern technology. 

The computer is making it possible to 
— send men and satellites into space 

— make significant strides in medical research 

— add several billions of dollars to our revenue through 
improved tax administration 
— administer the huge and complex social security and 
medicare programs 

-—manage a multi-billion dollar defense logistics system 
— speed the issuance of G.I. insurance dividends, at 
much less cost 

— save lives through better search and rescue operations 
— harness atomic energy for peaceful uses 

- — design better but less costly highways and structures. 
In short, computers arc enabling us to achieve progress 
and benefits which a decade ago were beyond our grasp. 

The technology is available. Its potential for good 
has been amply demonstrated, hut it remains to be tapped 
in fuller measure. 

I am determined that we take advantage of this tech- 
nology by using it imaginatively to accomplish worthwhile 
purposes. 

I tlieiefore want every agency head to give thorough 
study to new ways in which the electronic computer 
might be used to 

■ — provide better serv ice to the public 

— improv e agency per Tormain c 
— reduce costs. 

But, as we use computers to achieve these benefits, 1 
want these activities managed at the lowest possible cost. 
At the piesent time, the Federal Government 
— uses 2,600 computers 

— employs 7 1 ,000 p< ople in this at tiv il y 



— spends over $2 billion annually to acquire and op- 
erate this equipment, including special military type 
computers. 

Clearly, we must devote our best efforts to managing 
this large investment wisely and with the least cost. 

I approved a blueprint for action when I approved the 
Bureau of the Budget “Report on Management of ADI 5 
in the Government.” 

The Congress recognized this need when it enacted 
Public Law 89 306 (the Brooks Bill) last October. This 
legislation provided specific authorities lo 

— the General Set vices Administration, for the pro- 
curement, utilization and disposition of automatic 
data processing equipment 

— the Department of Commerce, for the development 
of data processing standards and the provision of 
assistance to agencies in designing computer-based 
systems 

— the Bureau of the Budget, for cxcrci ing policy and 
fiscal control over the implementation of these 
authoiities. 

These agencies aie seeking actively to put into effect 
ways for improving and reducing the cost of tins huge and 
complex operation. 

In my Budget Message for 1967 I told the Congress 
of my intent to make sure that this huge investment is 
managed efficiently. 

The Federal Government must give priority attention 

to 

— establishing better and more effective procurement 
methods 

- — making fuller use of existing facilities through shar- 
ing and joint-use arrangements before acquiring 
additional equipment 

— re-utilizing excess equipment whenever feasible 

— achieving, with industry cooperation, greater com- 
patibility of equipment. 

I expect all agencies to cooperate fully with the Bureau 
of the Budget, the General Services Administration, and 
the Department of Commerce in accomplishing these 
objectives. 

I want the Director of the Bureau of the Budget to 
report to me on December 3 1, 1966, and every six months 
thereafter, on the progress that is being made throughout 
the Federal Government in improving the management 
of this very important tei luiology. 
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'ALSO WCL'JDtS OTHER OESJC.’XJD SHORE ACTIVITIES, 

?:ot si;cv;a c.*i the ciust, racn ahs under the command 
or supervision c? ?:a>: y cf the oscAKiZAiiCiis depicted. 



APPENDIX E 



DIRECTIVES GOVERNING ACTIONS OF ADPESO 



Bureau of Budget Circulars 

o A- 27, Policies and Responsibilities of the Sharing and Electronic Computer 
Time and Services in the Executive Branch 

o A- 54, Policies on .Selection and Acquisition of Automatic Data Processing 
(ADP) Equipment 

o A- 61, Guidelines for Appraising Agency Practices in the Management of 
Automatic Data Processing (ADP) Equipment in Federal Agencies 

o A- 94, Discount Rates and Procedures. to Be Used Evaluating Deferred Costs 
and Benefits 



Department of Defense Instructions 



e 4160. 19, DOD Automatic Data Processing Equipment Reutilization Program, 
May 8, 1969 

o 7041. 3, Economic Analysis of Proposed Department of Defense Investments, 
February 28, 1969 

o 5100.40, Responsibilities for the Administration of Automatic Data Processing- 
Equipment Program, September 28, 1963 

o 4160. 1, Nonexcess Personal Property to be Sold or Exchanged for Replace- 
ment Purposes, August 10, 1966 

© 4160. 21, Department of Defense Personal Property Disposal Program, 

April 28, 1967 
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o 4140. 34, Department of Defense Personal Property Utilization Program, 
September 5, 1968 



Department of t he Navy Instructions 

o Comptroller of die Navy Instruction 7000. 28, Economic Analysis of Proposed 
Department of the Navy Investments, April 19, 1967 

0 Secretary of die Navy Instruction 10462. 7B, Department of the Navy Automatic 
Data Processing Program. March 11, 1966 
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APPENDIX F 

AUTOMATIC DATA PROCESSING SYSTEM SPECIFICATIONS 

CONTENTS 



Part I - Supporting Information 

Section I - Justification for Acquisition of ADP System 

1-1 History/ 

1-2 System Justification 

1- 3 Budgeting Estimates and Limitations 
Section 2 - Workload Description 

2- 1 Representative Programs 

2- 2 Derivation of Representative Programs 
Section 3 - Explanation of Requirements 

3- 1 Justification of Mandatory Requirements 

3-2 Statement of Desirable Features 

3-3 Statements of Value 

Part II - ADP System Specifications 

Section 1 - Format and Content of ADP System Specifications 

ATTACHMENT I 

Chapter 1 - Background and Operational Plans 
1- 1 Background 

1- 2 Operational Plans 
Chapter 2 - Requirements 

2- 1 System Mandatory Requirements 
2-2 Desirable Features 

Chapter 3 - Explanation of Terms and Abbreviations 

ATTACHMENT II 

Chapter 1 - System Questionnaire 
Chapter 2 - Timing Information 
2-1 General Information 
2-2 Program Description 
2-3 Extension Factors and Timing Tables 
2-4 Benchmark Demonstration Instructions 
Section 2 - Benchmark Package 

SOURCE: U. S. , Department of the Navy, Specification, Selection, an d 
Acquisition of Automatic Data Processing Eq ui pment , Secretary of the Navy. 
Instruction 10462.13, Department of the Navy, Washington, D.C., 20390, p. B-2. 
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APPENDIX G 



COMPUTER SYSTEMS IN SCERT LIBRARY 



EMR 

6000 Series 



Burroughs 
100 Series 
200 Series 
300 Scries 
2500 
3500 
5500 
6500 



Control Data 

1 60A 

1604 A 

1700 

3100 

3200 

3300 

3400 

3500 

3600 

3800 

6400 

6500 

6600 



General Electric 
115 
120 
205 
215 
225 
235 
415 
425 
435 
625 
635 



Honeywell 

110 

120 

125 

200 

400 

800 

1200 

1250 

1400 
1800 
2200 
3200 
4200 
8200 

IBM 

360/20 

360/25 

360/30 

360/40 

360/44 

360/50 

360/65 

360/67 

360/75 

360/85 

360/195 

1 130 

1401 
1410 
1440 
1620 
7010 

7040 Series 
705 

7070 Series 
7080 

7090 Series 

NCR 

315 

3 1 5 RMC 
CENTURY- 100 
CENTURY-200 

1CL 

1900 Series 
System 4 Series 



1 )h'.i!al Equipment 
PDP- 6 
PDP- 8 
PDP- 8i 
PDP- 10 



Philco 
2000 Series 

RCA 

301 

501 

601 

3301 

70/15 

70/25 

70/35 

70/45 

70/46 

70/55 

70/60 

UNI VAC 
III 
418 
418-111 
490 Series 

1004 Series 

1005 
1050 
1 106 

1 107 

1 108 
9200 
9300 
9400 



SDS 
920 
925 
930 
9300 
Sigma 2 
Sigma 5 
Sigma 7 
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APPENDIX H 



PRODUCTION OF OUTPUT REPORTS 



SCERT has been designed to produce a complete scries of "standai'd” output 
reports but also to be readily adaptable to the production of nonstandard or special 
purpose reports. The reports of the standard set are all programmed as optional, 
and the typical use of the program calls for just those reports needed for one’s 
particular analysis. A brief description of the standard reports: 

Computer Complement Report - Portrays the exact configuration that was 
simulated and provides certain basic cost data about that configuration. 

Central Processor Utilization - Summarizes for each scheduled run the 
projected running time, set-up time, and horizontal memory utilization. 

Programming Requirements - Projects for each run an implementation- oriented 
by-product of the simulation by summarizing the number of steps to be 
programmed and an estimate of the number of programmer man months 
required. 

Application Summary - If the processing requirements simulated can be 
broken into discreet application areas, this report summarizes running 
time and programming effort for each of these areas. 

Computer Capabilities - Tabulates for each scheduled run simulated break- 
out of thru- put requirements and then a total for the entire simulation. It is 
used primarily to pinpoint critical hardware areas. 

Cost Summary - Primary cost output which relates the projected utilization of 
the computer to lease, purchase, and maintenance cost considerations. 
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Real-time Analysis - If random real-time processing was involved, a series of 
reports are produced to portray the impact of such processing on the computer 
and the response expected of the computer for such processing. The four 
standard reports are: 

Event Processing Analysis - Analyzes each random event in terms of 
its unique thru- put. 

Hardware Utilization - Analyzes every potential queue present in the 
hardware complex. 

Systems Response - Reflects the expected 95th and 99lh percentile of 
computer and communications network response. 

. Memory Requirements - Outlines the unique memory required for each 

random event and the expected and worst case background requirements. 

Multiprogram Run Schedule - Produced whenever multiprogramming has been 
simulated and reflects those runs which have been scheduled by the simulator to 
operate concurrently. 

Detail Analysis - Produced for each scheduled run or real-time event simulated 
and serves as backup for all other reports. It precisely portrays the utilization, 
timing, and memory requirements derived by the simulation and presimulation 
algorithms for all components making up the run. 

Summary Analysis - Summarizes, for each configuration simulated, the 
performance and cost/performance derived. 

System Documentation - A by-product report which produces a standard form of 
documentation of all processing requirements. It is especially useful in securing 
responsive, realistic hardware bids from computer manufacturers. 
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SAMPLE FORMAT FOR DISCOUNTING DEFERRED COSTS AND BENEFITS 



Assume a ten-year program which will commit the Government to 
the stream of expenditures appearing in column (2) of the 
table below and which will result in a series of benefits 
appearing in column (3) . The discount factor for a 10 percent 
discount rate is presented in column (4) . Present value cost 
for each of the ten years is calculated by multiplying 
column (2) by column (4); present value benefit for each of 
the ten years is calculated by multiplying column (3) by 
column (4) . Present value costs and benefits are presented 
in columns (5) and (6) , respectively. 



Year of 
Operation 


Expected 

Yearly 

Cost 


Expected 

Yearly 

Benefit 


Discount 
Factor for 
10 Percent 


Present 

Value 

Cost 

[Col. ( 2 ) x 
Col. (4) ] 


Present 
Value 
Benefit 
[Col. ( 3 ) x 
Col . (4) ] 


(1) 


(2) 


(3) 


(4) 


(5) 


(6) 


1 


$10 


$ 0 


0 .909 


$9.1 


$ 0.0 


2 


20 


0 


0 .826 


16.5 


0.0 


3 


30 


5 


0-751 


22.5 


3.8 


4 


30 


10 


0.683 


20.5 


6.8 


5 


20 


30 


0.621 


12 .4 


18.6 


6 


10 


40 


0.564 


5 . 6 


22.6 


7 


5 


40 


0.513 


2.6 


20.5 


8 


5 


40 


0.467 


2 . 3 


18.7 


9 


5 


4 0 


0.424 


2.1 


17.0 


10 


5 


25 


0.386 


1.9 


9.7 










$95.5 


$117.7 


The sum of 


column 


(5) is present value 


cost : $95 


.5 . 



The sum of column (6) is present value benefit: $117.7. 

Present value net benefit is the difference between present 
value total benefit and present value total cost: 

$117.7 - $95.5 = $22.2. 

The benefit-cost ratio is 117.7/95.5 = 1.23. 

NOTE: For more difficult discounting problems, a recom- 

mended reference is Princi ples of Engineerin g 
Economy , by Eugene L. Grant and W. G. Ireson, 

Ronald Press Company, 1960. 
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